2022-09-27T15:24:18.5205326Z Requested labels: linux.2xlarge
2022-09-27T15:24:18.5205401Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/85462/merge
2022-09-27T15:24:18.5205429Z Waiting for a runner to pick up this job...
2022-09-27T15:24:22.5125153Z Job is about to start running on the runner: i-01f41056b46f41785 (organization)
2022-09-27T15:24:27.3325323Z Current runner version: '2.296.2'
2022-09-27T15:24:27.3330565Z Runner name: 'i-01f41056b46f41785'
2022-09-27T15:24:27.3331100Z Runner group name: 'Default'
2022-09-27T15:24:27.3331687Z Machine name: 'ip-10-0-1-206'
2022-09-27T15:24:27.3333601Z ##[group]GITHUB_TOKEN Permissions
2022-09-27T15:24:27.3334277Z Actions: read
2022-09-27T15:24:27.3334593Z Checks: read
2022-09-27T15:24:27.3334871Z Contents: read
2022-09-27T15:24:27.3335188Z Deployments: read
2022-09-27T15:24:27.3335537Z Discussions: read
2022-09-27T15:24:27.3335851Z Issues: read
2022-09-27T15:24:27.3336119Z Metadata: read
2022-09-27T15:24:27.3336463Z Packages: read
2022-09-27T15:24:27.3336768Z Pages: read
2022-09-27T15:24:27.3337072Z PullRequests: read
2022-09-27T15:24:27.3337437Z RepositoryProjects: read
2022-09-27T15:24:27.3337787Z SecurityEvents: read
2022-09-27T15:24:27.3338072Z Statuses: read
2022-09-27T15:24:27.3338386Z ##[endgroup]
2022-09-27T15:24:27.3341331Z Secret source: None
2022-09-27T15:24:27.3341906Z Prepare workflow directory
2022-09-27T15:24:27.5503334Z Prepare all required actions
2022-09-27T15:24:27.5662220Z Getting action download info
2022-09-27T15:24:27.7968390Z Download action repository 'pytorch/pytorch@master' (SHA:01dbbeeeb5ab7ede28e333982e98713282a0e4b8)
2022-09-27T15:24:31.0189488Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-09-27T15:24:31.1474176Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886)
2022-09-27T15:24:31.3908287Z Getting action download info
2022-09-27T15:24:31.5675602Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-09-27T15:24:31.7608526Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-09-27T15:24:31.7610430Z ##[group] Inputs
2022-09-27T15:24:31.7610740Z   build-environment: linux-focal-py3.7-gcc7
2022-09-27T15:24:31.7611556Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "docs_test", shard: 1, num_shards: 1,  runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
]}

2022-09-27T15:24:31.7612439Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:24:31.7612807Z   sync-tag: 
2022-09-27T15:24:31.7613016Z ##[endgroup]
2022-09-27T15:24:31.7613499Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 2, 2, linux.2xlarge)
2022-09-27T15:24:31.8289908Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-09-27T15:24:31.8290220Z with:
2022-09-27T15:24:31.8290447Z   submodules: recursive
2022-09-27T15:24:31.8290667Z   fetch-depth: 0
2022-09-27T15:24:31.8290856Z env:
2022-09-27T15:24:31.8291063Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:24:31.8291286Z ##[endgroup]
2022-09-27T15:24:31.8502582Z ##[group]Run retry () {
2022-09-27T15:24:31.8502845Z retry () {
2022-09-27T15:24:31.8503109Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-09-27T15:24:31.8503360Z }
2022-09-27T15:24:31.8503569Z echo "${GITHUB_WORKSPACE}"
2022-09-27T15:24:31.8504069Z if [ -z "${NO_SUDO}" ]; then
2022-09-27T15:24:31.8504331Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:24:31.8504570Z else
2022-09-27T15:24:31.8504789Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:24:31.8505028Z fi
2022-09-27T15:24:31.8505327Z mkdir "${GITHUB_WORKSPACE}"
2022-09-27T15:24:31.8520777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:24:31.8521058Z env:
2022-09-27T15:24:31.8521272Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:24:31.8521480Z   NO_SUDO: 
2022-09-27T15:24:31.8521686Z ##[endgroup]
2022-09-27T15:24:31.8687678Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:24:33.6058561Z ##[group]Run malfet/checkout@silent-checkout
2022-09-27T15:24:33.6058849Z with:
2022-09-27T15:24:33.6059084Z   ref: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:24:33.6059320Z   fetch-depth: 0
2022-09-27T15:24:33.6059524Z   submodules: recursive
2022-09-27T15:24:33.6059747Z   quiet-checkout: true
2022-09-27T15:24:33.6059980Z   repository: pytorch/pytorch
2022-09-27T15:24:33.6060373Z   token: ***
2022-09-27T15:24:33.6060581Z   ssh-strict: true
2022-09-27T15:24:33.6060810Z   persist-credentials: true
2022-09-27T15:24:33.6061032Z   clean: true
2022-09-27T15:24:33.6061225Z   lfs: false
2022-09-27T15:24:33.6061444Z   set-safe-directory: true
2022-09-27T15:24:33.6061663Z env:
2022-09-27T15:24:33.6061853Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:24:33.6062070Z ##[endgroup]
2022-09-27T15:24:33.7172364Z Syncing repository: pytorch/pytorch
2022-09-27T15:24:33.7173832Z ##[group]Getting Git version info
2022-09-27T15:24:33.7174351Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:24:33.7175196Z [command]/usr/bin/git version
2022-09-27T15:24:33.7175595Z git version 2.37.1
2022-09-27T15:24:33.7176268Z ##[endgroup]
2022-09-27T15:24:33.7188911Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/154f859a-277d-4ff6-8dcb-7e9d9cf316c4' before making global git config changes
2022-09-27T15:24:33.7189361Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T15:24:33.7189865Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:24:33.7216571Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:24:33.7219980Z ##[group]Initializing the repository
2022-09-27T15:24:33.7224393Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:24:33.7252769Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-09-27T15:24:33.7253572Z hint: is subject to change. To configure the initial branch name to use in all
2022-09-27T15:24:33.7254176Z hint: of your new repositories, which will suppress this warning, call:
2022-09-27T15:24:33.7254458Z hint: 
2022-09-27T15:24:33.7254770Z hint: 	git config --global init.defaultBranch <name>
2022-09-27T15:24:33.7255146Z hint: 
2022-09-27T15:24:33.7255481Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-09-27T15:24:33.7256001Z hint: 'development'. The just-created branch can be renamed via this command:
2022-09-27T15:24:33.7256276Z hint: 
2022-09-27T15:24:33.7256647Z hint: 	git branch -m <name>
2022-09-27T15:24:33.7257055Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-09-27T15:24:33.7264163Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-09-27T15:24:33.7292378Z ##[endgroup]
2022-09-27T15:24:33.7292805Z ##[group]Disabling automatic garbage collection
2022-09-27T15:24:33.7297090Z [command]/usr/bin/git config --local gc.auto 0
2022-09-27T15:24:33.7323204Z ##[endgroup]
2022-09-27T15:24:33.7330381Z ##[group]Setting up auth
2022-09-27T15:24:33.7330762Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T15:24:33.7359524Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T15:24:33.7604130Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T15:24:33.7631666Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T15:24:33.7873860Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:24:33.7908886Z ##[endgroup]
2022-09-27T15:24:33.7909327Z ##[group]Fetching the repository
2022-09-27T15:24:33.7914929Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-09-27T15:25:21.2842521Z [command]/usr/bin/git rev-parse --verify --quiet 52424e2bf38e454d535881fed9628d3e20f4f944^{object}
2022-09-27T15:25:21.2877364Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:25:22.4676351Z ##[endgroup]
2022-09-27T15:25:22.4676813Z ##[group]Determining the checkout info
2022-09-27T15:25:22.4677865Z ##[endgroup]
2022-09-27T15:25:22.4678422Z ##[group]Checking out the ref
2022-09-27T15:25:22.4683306Z [command]/usr/bin/git checkout --quiet --force 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:25:23.6824785Z ##[endgroup]
2022-09-27T15:25:23.6825219Z ##[group]Setting up auth for fetching submodules
2022-09-27T15:25:23.6830888Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:25:23.6876543Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-09-27T15:25:23.6908096Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-09-27T15:25:23.6936753Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-09-27T15:25:23.6961540Z ##[endgroup]
2022-09-27T15:25:23.6961946Z ##[group]Fetching submodules
2022-09-27T15:25:23.6965839Z [command]/usr/bin/git submodule sync --recursive
2022-09-27T15:25:23.7220419Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-09-27T15:25:23.7465618Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-09-27T15:25:23.7467814Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-09-27T15:25:23.7469729Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-09-27T15:25:23.7471759Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-09-27T15:25:23.7473916Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-09-27T15:25:23.7476508Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-09-27T15:25:23.7478589Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-09-27T15:25:23.7481101Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-09-27T15:25:23.7483763Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-09-27T15:25:23.7486381Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-09-27T15:25:23.7489172Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-09-27T15:25:23.7491973Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-09-27T15:25:23.7494956Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-09-27T15:25:23.7498056Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-09-27T15:25:23.7501247Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-09-27T15:25:23.7504621Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-09-27T15:25:23.7508133Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-09-27T15:25:23.7511677Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:25:23.7515199Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-09-27T15:25:23.7518986Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-09-27T15:25:23.7522755Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-09-27T15:25:23.7526821Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-09-27T15:25:23.7530703Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-09-27T15:25:23.7534771Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-09-27T15:25:23.7538924Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-09-27T15:25:23.7543200Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-09-27T15:25:23.7547895Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-09-27T15:25:23.7552292Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-09-27T15:25:23.7556954Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-09-27T15:25:23.7561562Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-09-27T15:25:23.7566328Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-09-27T15:25:23.7571126Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-09-27T15:25:23.7576119Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-09-27T15:25:23.7581213Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-09-27T15:25:23.7586589Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-09-27T15:25:23.7591850Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-09-27T15:25:23.7597189Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-09-27T15:25:23.7602554Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-09-27T15:25:23.7608185Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-09-27T15:25:23.7614002Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-09-27T15:25:23.7619818Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-09-27T15:25:23.7643199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-09-27T15:25:24.0646170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-09-27T15:25:24.3347953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-09-27T15:25:24.5808999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-09-27T15:25:24.8940545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-09-27T15:25:25.2087461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-09-27T15:25:27.3402457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-09-27T15:25:32.9393696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-09-27T15:25:33.3359138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-09-27T15:25:33.8656050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-09-27T15:25:35.5309405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-09-27T15:25:36.5981033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-09-27T15:25:37.7676449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-09-27T15:25:46.6731744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-09-27T15:25:47.3560008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-09-27T15:25:48.4537075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-09-27T15:25:49.5797527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-09-27T15:25:49.8109754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-09-27T15:25:50.2440172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-09-27T15:25:50.5259356Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-09-27T15:25:51.4710124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-09-27T15:25:51.8814423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-09-27T15:25:52.1112066Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-09-27T15:25:52.4129448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-09-27T15:25:53.7217256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-09-27T15:25:54.2557523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-09-27T15:25:54.7737633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-09-27T15:26:00.5131010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-09-27T15:26:02.0549035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-09-27T15:26:02.6441348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-09-27T15:26:02.8898572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-09-27T15:26:08.6255355Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-09-27T15:26:08.8336923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-09-27T15:26:09.0442746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-09-27T15:26:09.7872787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-09-27T15:26:10.0017574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-09-27T15:26:10.3074370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-09-27T15:26:10.6183079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-09-27T15:26:11.1641908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-09-27T15:26:13.1216981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-09-27T15:26:13.5886608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-09-27T15:26:15.7666698Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-09-27T15:26:15.7761451Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-09-27T15:26:15.7834154Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-09-27T15:26:15.8033961Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-09-27T15:26:15.8231560Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-09-27T15:26:15.8567413Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-09-27T15:26:16.3990588Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-09-27T15:26:16.4181587Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:26:16.5079916Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-09-27T15:26:16.5384981Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-09-27T15:26:16.7996258Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-09-27T15:26:17.1687886Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-09-27T15:26:17.3910619Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-09-27T15:26:17.4330226Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030'
2022-09-27T15:26:17.4343173Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:17.4345683Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:17.4348233Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:17.4351074Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:17.4372840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-09-27T15:26:18.2832246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-09-27T15:26:18.8007498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-09-27T15:26:19.7795372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-09-27T15:26:20.0936941Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-09-27T15:26:20.1849092Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-09-27T15:26:20.2370036Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-09-27T15:26:20.2459412Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-09-27T15:26:20.3228645Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-09-27T15:26:20.3529251Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-09-27T15:26:20.3609228Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-09-27T15:26:20.3963341Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-09-27T15:26:20.4177149Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-09-27T15:26:20.4589781Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-09-27T15:26:20.4689009Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-09-27T15:26:20.4701941Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:20.4722153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-09-27T15:26:27.1524214Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-09-27T15:26:27.1538518Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:27.1561302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-09-27T15:26:33.9895086Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-09-27T15:26:33.9990384Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-09-27T15:26:34.0118359Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-09-27T15:26:34.0965842Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-09-27T15:26:34.0979090Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:34.0981435Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:34.1004247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-09-27T15:26:35.3192462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-09-27T15:26:36.4972958Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-09-27T15:26:36.5454742Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-09-27T15:26:36.5630498Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-09-27T15:26:36.5753481Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-09-27T15:26:36.6709062Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-09-27T15:26:36.8900763Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-09-27T15:26:36.8927402Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:36.8928734Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:36.8952531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-09-27T15:26:37.2835261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-09-27T15:26:38.0515949Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:26:38.0793317Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-09-27T15:26:38.0927759Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-09-27T15:26:38.0940472Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:38.0960675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-09-27T15:26:39.7346081Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-09-27T15:26:39.7363786Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:39.7365312Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:39.7388371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-09-27T15:26:41.1325091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-09-27T15:26:41.8748888Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-09-27T15:26:41.9322789Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-09-27T15:26:41.9334667Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:41.9355883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-09-27T15:26:42.1429335Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:26:42.1510760Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-09-27T15:26:42.3812528Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-09-27T15:26:42.3831716Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:42.3833277Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:42.3856212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-09-27T15:26:42.7829559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-09-27T15:26:43.7570745Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-09-27T15:26:43.8188365Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-09-27T15:26:43.8263470Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-09-27T15:26:43.8358492Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-09-27T15:26:43.8650204Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-09-27T15:26:43.8727867Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-09-27T15:26:43.8976872Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-09-27T15:26:43.9059912Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-09-27T15:26:43.9455018Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-09-27T15:26:44.0423607Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-09-27T15:26:44.0655795Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-09-27T15:26:44.0671097Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:44.0672865Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:44.0675958Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:44.0678781Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:44.0699818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-09-27T15:26:45.0117116Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-09-27T15:26:45.2746749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-09-27T15:26:46.3768769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-09-27T15:26:47.1840501Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-09-27T15:26:47.1970762Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-09-27T15:26:47.2556388Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-09-27T15:26:47.2800969Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-09-27T15:26:47.2812973Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:47.2834122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-09-27T15:26:47.4959165Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:26:47.6164226Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-09-27T15:26:47.6192060Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-09-27T15:26:47.6448487Z Entering 'android/libs/fbjni'
2022-09-27T15:26:47.6482065Z Entering 'third_party/FP16'
2022-09-27T15:26:47.6516592Z Entering 'third_party/FXdiv'
2022-09-27T15:26:47.6550264Z Entering 'third_party/NNPACK'
2022-09-27T15:26:47.6583801Z Entering 'third_party/QNNPACK'
2022-09-27T15:26:47.6618228Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:26:47.6652878Z Entering 'third_party/XNNPACK'
2022-09-27T15:26:47.6696364Z Entering 'third_party/benchmark'
2022-09-27T15:26:47.6730487Z Entering 'third_party/cpuinfo'
2022-09-27T15:26:47.6764893Z Entering 'third_party/cub'
2022-09-27T15:26:47.6799380Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:26:47.6837790Z Entering 'third_party/cutlass'
2022-09-27T15:26:47.6877393Z Entering 'third_party/eigen'
2022-09-27T15:26:47.6913668Z Entering 'third_party/fbgemm'
2022-09-27T15:26:47.6947323Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:47.6979581Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:47.7012777Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:47.7046532Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:47.7080404Z Entering 'third_party/flatbuffers'
2022-09-27T15:26:47.7115951Z Entering 'third_party/fmt'
2022-09-27T15:26:47.7149105Z Entering 'third_party/foxi'
2022-09-27T15:26:47.7182137Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:26:47.7216345Z Entering 'third_party/gloo'
2022-09-27T15:26:47.7250653Z Entering 'third_party/googletest'
2022-09-27T15:26:47.7285747Z Entering 'third_party/ideep'
2022-09-27T15:26:47.7319014Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:47.7353126Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:47.7393456Z Entering 'third_party/ios-cmake'
2022-09-27T15:26:47.7427146Z Entering 'third_party/ittapi'
2022-09-27T15:26:47.7459474Z Entering 'third_party/kineto'
2022-09-27T15:26:47.7493561Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:47.7526871Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:47.7561171Z Entering 'third_party/nccl/nccl'
2022-09-27T15:26:47.7594290Z Entering 'third_party/neon2sse'
2022-09-27T15:26:47.7629506Z Entering 'third_party/nlohmann'
2022-09-27T15:26:47.7664366Z Entering 'third_party/onnx'
2022-09-27T15:26:47.7708856Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:47.7742470Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:47.7779491Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:26:47.7812843Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:47.7850867Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:47.7884236Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:47.7918166Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:47.7956279Z Entering 'third_party/pocketfft'
2022-09-27T15:26:47.7990273Z Entering 'third_party/protobuf'
2022-09-27T15:26:47.8026690Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:47.8059548Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:47.8095082Z Entering 'third_party/psimd'
2022-09-27T15:26:47.8128831Z Entering 'third_party/pthreadpool'
2022-09-27T15:26:47.8162503Z Entering 'third_party/pybind11'
2022-09-27T15:26:47.8196572Z Entering 'third_party/python-enum'
2022-09-27T15:26:47.8230504Z Entering 'third_party/python-peachpy'
2022-09-27T15:26:47.8263590Z Entering 'third_party/python-six'
2022-09-27T15:26:47.8297894Z Entering 'third_party/sleef'
2022-09-27T15:26:47.8331522Z Entering 'third_party/tbb'
2022-09-27T15:26:47.8366736Z Entering 'third_party/tensorpipe'
2022-09-27T15:26:47.8401101Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:47.8434749Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:47.8467929Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:47.8500945Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:47.8534326Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:47.8569559Z Entering 'third_party/zstd'
2022-09-27T15:26:47.8610312Z ##[endgroup]
2022-09-27T15:26:47.8612367Z ##[group]Persisting credentials for submodules
2022-09-27T15:26:47.8618735Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-09-27T15:26:47.8869656Z Entering 'android/libs/fbjni'
2022-09-27T15:26:47.8902287Z Entering 'third_party/FP16'
2022-09-27T15:26:47.8935468Z Entering 'third_party/FXdiv'
2022-09-27T15:26:47.8968368Z Entering 'third_party/NNPACK'
2022-09-27T15:26:47.9001611Z Entering 'third_party/QNNPACK'
2022-09-27T15:26:47.9036032Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:26:47.9068995Z Entering 'third_party/XNNPACK'
2022-09-27T15:26:47.9112626Z Entering 'third_party/benchmark'
2022-09-27T15:26:47.9145327Z Entering 'third_party/cpuinfo'
2022-09-27T15:26:47.9178446Z Entering 'third_party/cub'
2022-09-27T15:26:47.9211501Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:26:47.9248894Z Entering 'third_party/cutlass'
2022-09-27T15:26:47.9288574Z Entering 'third_party/eigen'
2022-09-27T15:26:47.9323093Z Entering 'third_party/fbgemm'
2022-09-27T15:26:47.9356326Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:47.9388254Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:47.9421526Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:47.9454596Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:47.9487731Z Entering 'third_party/flatbuffers'
2022-09-27T15:26:47.9522826Z Entering 'third_party/fmt'
2022-09-27T15:26:47.9556844Z Entering 'third_party/foxi'
2022-09-27T15:26:47.9590936Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:26:47.9624500Z Entering 'third_party/gloo'
2022-09-27T15:26:47.9657156Z Entering 'third_party/googletest'
2022-09-27T15:26:47.9690973Z Entering 'third_party/ideep'
2022-09-27T15:26:47.9723575Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:47.9757575Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:47.9796414Z Entering 'third_party/ios-cmake'
2022-09-27T15:26:47.9829544Z Entering 'third_party/ittapi'
2022-09-27T15:26:47.9862480Z Entering 'third_party/kineto'
2022-09-27T15:26:47.9896605Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:47.9929320Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:47.9964206Z Entering 'third_party/nccl/nccl'
2022-09-27T15:26:47.9997274Z Entering 'third_party/neon2sse'
2022-09-27T15:26:48.0031176Z Entering 'third_party/nlohmann'
2022-09-27T15:26:48.0064975Z Entering 'third_party/onnx'
2022-09-27T15:26:48.0109240Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.0140777Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.0175306Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:26:48.0207714Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:48.0245604Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.0278083Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.0310989Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.0347401Z Entering 'third_party/pocketfft'
2022-09-27T15:26:48.0380326Z Entering 'third_party/protobuf'
2022-09-27T15:26:48.0416691Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:48.0448970Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:48.0483810Z Entering 'third_party/psimd'
2022-09-27T15:26:48.0516934Z Entering 'third_party/pthreadpool'
2022-09-27T15:26:48.0549595Z Entering 'third_party/pybind11'
2022-09-27T15:26:48.0581973Z Entering 'third_party/python-enum'
2022-09-27T15:26:48.0615476Z Entering 'third_party/python-peachpy'
2022-09-27T15:26:48.0648237Z Entering 'third_party/python-six'
2022-09-27T15:26:48.0680473Z Entering 'third_party/sleef'
2022-09-27T15:26:48.0714724Z Entering 'third_party/tbb'
2022-09-27T15:26:48.0749909Z Entering 'third_party/tensorpipe'
2022-09-27T15:26:48.0782622Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:48.0815404Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:48.0847930Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:48.0882241Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:48.0914686Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.0949225Z Entering 'third_party/zstd'
2022-09-27T15:26:48.0993849Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-09-27T15:26:48.1240948Z Entering 'android/libs/fbjni'
2022-09-27T15:26:48.1271754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-09-27T15:26:48.1285549Z Entering 'third_party/FP16'
2022-09-27T15:26:48.1317322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-09-27T15:26:48.1330999Z Entering 'third_party/FXdiv'
2022-09-27T15:26:48.1362291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-09-27T15:26:48.1375937Z Entering 'third_party/NNPACK'
2022-09-27T15:26:48.1406204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-09-27T15:26:48.1419901Z Entering 'third_party/QNNPACK'
2022-09-27T15:26:48.1450208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-09-27T15:26:48.1463769Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:26:48.1495236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-09-27T15:26:48.1509458Z Entering 'third_party/XNNPACK'
2022-09-27T15:26:48.1540243Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-09-27T15:26:48.1563177Z Entering 'third_party/benchmark'
2022-09-27T15:26:48.1593871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:26:48.1607480Z Entering 'third_party/cpuinfo'
2022-09-27T15:26:48.1638587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:26:48.1652880Z Entering 'third_party/cub'
2022-09-27T15:26:48.1683269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-09-27T15:26:48.1697341Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:26:48.1727707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-09-27T15:26:48.1746644Z Entering 'third_party/cutlass'
2022-09-27T15:26:48.1777822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-09-27T15:26:48.1799736Z Entering 'third_party/eigen'
2022-09-27T15:26:48.1830671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-09-27T15:26:48.1847113Z Entering 'third_party/fbgemm'
2022-09-27T15:26:48.1878515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-09-27T15:26:48.1891864Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:48.1923790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-09-27T15:26:48.1937520Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:48.1969517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:26:48.1983538Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:48.2014546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:26:48.2028323Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:48.2066677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-09-27T15:26:48.2074662Z Entering 'third_party/flatbuffers'
2022-09-27T15:26:48.2106541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-09-27T15:26:48.2122653Z Entering 'third_party/fmt'
2022-09-27T15:26:48.2154126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-09-27T15:26:48.2168472Z Entering 'third_party/foxi'
2022-09-27T15:26:48.2200432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-09-27T15:26:48.2214135Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:26:48.2245445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-09-27T15:26:48.2260117Z Entering 'third_party/gloo'
2022-09-27T15:26:48.2291103Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-09-27T15:26:48.2306384Z Entering 'third_party/googletest'
2022-09-27T15:26:48.2337337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:26:48.2352236Z Entering 'third_party/ideep'
2022-09-27T15:26:48.2383665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-09-27T15:26:48.2398212Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:48.2429456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-09-27T15:26:48.2444273Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:48.2475445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-09-27T15:26:48.2496663Z Entering 'third_party/ios-cmake'
2022-09-27T15:26:48.2527622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-09-27T15:26:48.2541609Z Entering 'third_party/ittapi'
2022-09-27T15:26:48.2573257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-09-27T15:26:48.2588320Z Entering 'third_party/kineto'
2022-09-27T15:26:48.2620283Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-09-27T15:26:48.2634600Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:48.2666839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-09-27T15:26:48.2681509Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:48.2714009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-09-27T15:26:48.2729899Z Entering 'third_party/nccl/nccl'
2022-09-27T15:26:48.2762750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-09-27T15:26:48.2777645Z Entering 'third_party/neon2sse'
2022-09-27T15:26:48.2808899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-09-27T15:26:48.2823413Z Entering 'third_party/nlohmann'
2022-09-27T15:26:48.2854404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-09-27T15:26:48.2870308Z Entering 'third_party/onnx'
2022-09-27T15:26:48.2902070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:26:48.2927310Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.2961030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:26:48.2975629Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.3007378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:26:48.3024116Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:26:48.3057401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-09-27T15:26:48.3071332Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:48.3102568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:26:48.3120221Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.3152706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:26:48.3166940Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.3197461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:26:48.3210941Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.3245115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:26:48.3263681Z Entering 'third_party/pocketfft'
2022-09-27T15:26:48.3295317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-09-27T15:26:48.3308848Z Entering 'third_party/protobuf'
2022-09-27T15:26:48.3339649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-09-27T15:26:48.3357000Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:48.3388462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:26:48.3401990Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:48.3433615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:26:48.3448579Z Entering 'third_party/psimd'
2022-09-27T15:26:48.3479378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-09-27T15:26:48.3492712Z Entering 'third_party/pthreadpool'
2022-09-27T15:26:48.3523619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-09-27T15:26:48.3536422Z Entering 'third_party/pybind11'
2022-09-27T15:26:48.3565966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:26:48.3580298Z Entering 'third_party/python-enum'
2022-09-27T15:26:48.3611383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-09-27T15:26:48.3624864Z Entering 'third_party/python-peachpy'
2022-09-27T15:26:48.3657791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-09-27T15:26:48.3672591Z Entering 'third_party/python-six'
2022-09-27T15:26:48.3705636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-09-27T15:26:48.3718064Z Entering 'third_party/sleef'
2022-09-27T15:26:48.3751038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-09-27T15:26:48.3765032Z Entering 'third_party/tbb'
2022-09-27T15:26:48.3794703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-09-27T15:26:48.3811565Z Entering 'third_party/tensorpipe'
2022-09-27T15:26:48.3844546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-09-27T15:26:48.3861057Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:48.3894879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:26:48.3910064Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:48.3942810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-09-27T15:26:48.3957588Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:48.3990317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-09-27T15:26:48.4004129Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:48.4035468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:26:48.4049123Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.4082919Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:26:48.4099178Z Entering 'third_party/zstd'
2022-09-27T15:26:48.4131130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-09-27T15:26:48.5346204Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-09-27T15:26:48.5616206Z Entering 'android/libs/fbjni'
2022-09-27T15:26:48.5649767Z Entering 'third_party/FP16'
2022-09-27T15:26:48.5685367Z Entering 'third_party/FXdiv'
2022-09-27T15:26:48.5719474Z Entering 'third_party/NNPACK'
2022-09-27T15:26:48.5755022Z Entering 'third_party/QNNPACK'
2022-09-27T15:26:48.5790239Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:26:48.5825751Z Entering 'third_party/XNNPACK'
2022-09-27T15:26:48.5867455Z Entering 'third_party/benchmark'
2022-09-27T15:26:48.5902715Z Entering 'third_party/cpuinfo'
2022-09-27T15:26:48.5935896Z Entering 'third_party/cub'
2022-09-27T15:26:48.5973541Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:26:48.6012440Z Entering 'third_party/cutlass'
2022-09-27T15:26:48.6052425Z Entering 'third_party/eigen'
2022-09-27T15:26:48.6089509Z Entering 'third_party/fbgemm'
2022-09-27T15:26:48.6124709Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:48.6160664Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:48.6196094Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:48.6230730Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:48.6267478Z Entering 'third_party/flatbuffers'
2022-09-27T15:26:48.6303454Z Entering 'third_party/fmt'
2022-09-27T15:26:48.6336247Z Entering 'third_party/foxi'
2022-09-27T15:26:48.6368344Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:26:48.6400873Z Entering 'third_party/gloo'
2022-09-27T15:26:48.6437924Z Entering 'third_party/googletest'
2022-09-27T15:26:48.6471173Z Entering 'third_party/ideep'
2022-09-27T15:26:48.6505576Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:48.6539352Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:48.6580302Z Entering 'third_party/ios-cmake'
2022-09-27T15:26:48.6614146Z Entering 'third_party/ittapi'
2022-09-27T15:26:48.6647574Z Entering 'third_party/kineto'
2022-09-27T15:26:48.6681330Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:48.6714852Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:48.6750482Z Entering 'third_party/nccl/nccl'
2022-09-27T15:26:48.6786478Z Entering 'third_party/neon2sse'
2022-09-27T15:26:48.6822258Z Entering 'third_party/nlohmann'
2022-09-27T15:26:48.6860495Z Entering 'third_party/onnx'
2022-09-27T15:26:48.6906980Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.6939556Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.6979090Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:26:48.7012698Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:48.7050883Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.7085957Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.7120506Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.7159506Z Entering 'third_party/pocketfft'
2022-09-27T15:26:48.7193464Z Entering 'third_party/protobuf'
2022-09-27T15:26:48.7233083Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:48.7267049Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:48.7302252Z Entering 'third_party/psimd'
2022-09-27T15:26:48.7338006Z Entering 'third_party/pthreadpool'
2022-09-27T15:26:48.7373174Z Entering 'third_party/pybind11'
2022-09-27T15:26:48.7407637Z Entering 'third_party/python-enum'
2022-09-27T15:26:48.7442565Z Entering 'third_party/python-peachpy'
2022-09-27T15:26:48.7476861Z Entering 'third_party/python-six'
2022-09-27T15:26:48.7510628Z Entering 'third_party/sleef'
2022-09-27T15:26:48.7545040Z Entering 'third_party/tbb'
2022-09-27T15:26:48.7581542Z Entering 'third_party/tensorpipe'
2022-09-27T15:26:48.7616198Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:48.7649712Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:48.7683007Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:48.7716593Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:48.7750242Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.7785716Z Entering 'third_party/zstd'
2022-09-27T15:26:48.7834677Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-09-27T15:26:48.8085732Z Entering 'android/libs/fbjni'
2022-09-27T15:26:48.8119292Z Entering 'third_party/FP16'
2022-09-27T15:26:48.8154268Z Entering 'third_party/FXdiv'
2022-09-27T15:26:48.8188414Z Entering 'third_party/NNPACK'
2022-09-27T15:26:48.8222060Z Entering 'third_party/QNNPACK'
2022-09-27T15:26:48.8256026Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:26:48.8290275Z Entering 'third_party/XNNPACK'
2022-09-27T15:26:48.8333481Z Entering 'third_party/benchmark'
2022-09-27T15:26:48.8367441Z Entering 'third_party/cpuinfo'
2022-09-27T15:26:48.8401748Z Entering 'third_party/cub'
2022-09-27T15:26:48.8436622Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:26:48.8476052Z Entering 'third_party/cutlass'
2022-09-27T15:26:48.8517437Z Entering 'third_party/eigen'
2022-09-27T15:26:48.8552597Z Entering 'third_party/fbgemm'
2022-09-27T15:26:48.8588412Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:26:48.8621920Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:26:48.8655299Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:26:48.8688762Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:26:48.8724026Z Entering 'third_party/flatbuffers'
2022-09-27T15:26:48.8758964Z Entering 'third_party/fmt'
2022-09-27T15:26:48.8793175Z Entering 'third_party/foxi'
2022-09-27T15:26:48.8826333Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:26:48.8860535Z Entering 'third_party/gloo'
2022-09-27T15:26:48.8895498Z Entering 'third_party/googletest'
2022-09-27T15:26:48.8930567Z Entering 'third_party/ideep'
2022-09-27T15:26:48.8964151Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:26:48.8998517Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:26:48.9038847Z Entering 'third_party/ios-cmake'
2022-09-27T15:26:48.9073738Z Entering 'third_party/ittapi'
2022-09-27T15:26:48.9108758Z Entering 'third_party/kineto'
2022-09-27T15:26:48.9144186Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:26:48.9178787Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:26:48.9215041Z Entering 'third_party/nccl/nccl'
2022-09-27T15:26:48.9250276Z Entering 'third_party/neon2sse'
2022-09-27T15:26:48.9285131Z Entering 'third_party/nlohmann'
2022-09-27T15:26:48.9320448Z Entering 'third_party/onnx'
2022-09-27T15:26:48.9364849Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.9399759Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.9436139Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:26:48.9471575Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:26:48.9509996Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:26:48.9544666Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:26:48.9578407Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:26:48.9616144Z Entering 'third_party/pocketfft'
2022-09-27T15:26:48.9650860Z Entering 'third_party/protobuf'
2022-09-27T15:26:48.9689605Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:26:48.9722804Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:26:48.9756989Z Entering 'third_party/psimd'
2022-09-27T15:26:48.9792276Z Entering 'third_party/pthreadpool'
2022-09-27T15:26:48.9826154Z Entering 'third_party/pybind11'
2022-09-27T15:26:48.9858670Z Entering 'third_party/python-enum'
2022-09-27T15:26:48.9892149Z Entering 'third_party/python-peachpy'
2022-09-27T15:26:48.9925953Z Entering 'third_party/python-six'
2022-09-27T15:26:48.9960400Z Entering 'third_party/sleef'
2022-09-27T15:26:48.9995432Z Entering 'third_party/tbb'
2022-09-27T15:26:49.0030869Z Entering 'third_party/tensorpipe'
2022-09-27T15:26:49.0065212Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:26:49.0098880Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:26:49.0132794Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:26:49.0167325Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:26:49.0200464Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:26:49.0236830Z Entering 'third_party/zstd'
2022-09-27T15:26:49.0279256Z ##[endgroup]
2022-09-27T15:26:49.0319844Z [command]/usr/bin/git log -1 --format='%H'
2022-09-27T15:26:49.0347862Z '52424e2bf38e454d535881fed9628d3e20f4f944'
2022-09-27T15:26:49.0471609Z Prepare all required actions
2022-09-27T15:26:49.0506264Z ##[group]Run ./.github/actions/setup-linux
2022-09-27T15:26:49.0506579Z env:
2022-09-27T15:26:49.0506848Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:49.0507151Z ##[endgroup]
2022-09-27T15:26:49.0728170Z ##[group]Run set -euo pipefail
2022-09-27T15:26:49.0728403Z set -euo pipefail
2022-09-27T15:26:49.0728600Z function get_ec2_metadata() {
2022-09-27T15:26:49.0728852Z   # Pulled from instance metadata endpoint for EC2
2022-09-27T15:26:49.0729209Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-09-27T15:26:49.0729506Z   category=$1
2022-09-27T15:26:49.0729739Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-09-27T15:26:49.0729961Z }
2022-09-27T15:26:49.0730162Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-09-27T15:26:49.0730442Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-09-27T15:26:49.0730809Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-09-27T15:26:49.0731057Z echo "system info $(uname -a)"
2022-09-27T15:26:49.0742126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:26:49.0742331Z env:
2022-09-27T15:26:49.0742510Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:49.0742703Z ##[endgroup]
2022-09-27T15:26:49.0820453Z ami-id: ami-096198a0bccc6bad4
2022-09-27T15:26:49.0869154Z instance-id: i-01f41056b46f41785
2022-09-27T15:26:49.0918529Z instance-type: c5.2xlarge
2022-09-27T15:26:49.0925384Z system info Linux ip-10-0-1-206.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-09-27T15:26:49.1043006Z ##[group]Run if systemctl is-active --quiet docker; then
2022-09-27T15:26:49.1043313Z if systemctl is-active --quiet docker; then
2022-09-27T15:26:49.1043595Z     echo "Docker daemon is running...";
2022-09-27T15:26:49.1043845Z else
2022-09-27T15:26:49.1044099Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-09-27T15:26:49.1044364Z fi
2022-09-27T15:26:49.1054815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:26:49.1055055Z env:
2022-09-27T15:26:49.1055262Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:49.1055485Z ##[endgroup]
2022-09-27T15:26:49.1096417Z Docker daemon is running...
2022-09-27T15:26:49.1187722Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:26:49.1188086Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:26:49.1188368Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:26:49.1188741Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-09-27T15:26:49.1189079Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-09-27T15:26:49.1199111Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:26:49.1199325Z env:
2022-09-27T15:26:49.1199484Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:49.1199682Z   AWS_RETRY_MODE: standard
2022-09-27T15:26:49.1199871Z   AWS_MAX_ATTEMPTS: 5
2022-09-27T15:26:49.1200056Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T15:26:49.1200246Z ##[endgroup]
2022-09-27T15:26:49.8392928Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-09-27T15:26:49.8393367Z Configure a credential helper to remove this warning. See
2022-09-27T15:26:49.8393947Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-09-27T15:26:49.8394146Z 
2022-09-27T15:26:49.8394270Z Login Succeeded
2022-09-27T15:26:49.8429810Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:26:49.8430222Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:26:49.8430627Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:26:49.8441953Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:26:49.8442273Z env:
2022-09-27T15:26:49.8442541Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:49.8442820Z ##[endgroup]
2022-09-27T15:26:49.8510858Z Prepare all required actions
2022-09-27T15:26:49.8511209Z Getting action download info
2022-09-27T15:26:50.0164386Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638)
2022-09-27T15:26:50.2162364Z ##[group]Run ./.github/actions/setup-ssh
2022-09-27T15:26:50.2162568Z with:
2022-09-27T15:26:50.2162887Z   github-secret: ***
2022-09-27T15:26:50.2163046Z env:
2022-09-27T15:26:50.2163222Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:50.2163405Z ##[endgroup]
2022-09-27T15:26:50.2279796Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-09-27T15:26:50.2280005Z with:
2022-09-27T15:26:50.2280730Z   GITHUB_TOKEN: ***
2022-09-27T15:26:50.2280915Z   activate-with-label: false
2022-09-27T15:26:50.2281185Z   label: with-ssh
2022-09-27T15:26:50.2281378Z   remove-existing-keys: true
2022-09-27T15:26:50.2281545Z env:
2022-09-27T15:26:50.2281715Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:50.2281899Z ##[endgroup]
2022-09-27T15:26:50.6534334Z Grabbing public ssh keys from https://github.com/kongzii.keys
2022-09-27T15:26:50.7416148Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-09-27T15:26:50.7429068Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-09-27T15:26:50.7450030Z Login using: ssh ec2-user@ec2-34-205-9-82.compute-1.amazonaws.com
2022-09-27T15:26:50.7507097Z Prepare all required actions
2022-09-27T15:26:50.7526084Z ##[group]Run ./.github/actions/pull-docker-image
2022-09-27T15:26:50.7526328Z with:
2022-09-27T15:26:50.7526699Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:26:50.7527044Z env:
2022-09-27T15:26:50.7527256Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:50.7527472Z ##[endgroup]
2022-09-27T15:26:50.7541141Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:26:50.7541446Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:26:50.7541746Z # ignore output since only exit code is used for conditional
2022-09-27T15:26:50.7542047Z # only pull docker image if it's not available locally
2022-09-27T15:26:50.7542375Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-09-27T15:26:50.7542711Z   retry docker pull "${DOCKER_IMAGE}"
2022-09-27T15:26:50.7542929Z fi
2022-09-27T15:26:50.7554234Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:26:50.7554483Z env:
2022-09-27T15:26:50.7554676Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:26:50.7555072Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:26:50.7555446Z ##[endgroup]
2022-09-27T15:26:51.0110496Z e66cf5fa0a4d4ed512901b12ccdab95cca946a29: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7
2022-09-27T15:26:51.0111273Z 675920708c8b: Pulling fs layer
2022-09-27T15:26:51.0112027Z 7249ff1ca2cc: Pulling fs layer
2022-09-27T15:26:51.0112391Z a1a4ad211d43: Pulling fs layer
2022-09-27T15:26:51.0112781Z dac0b0909394: Pulling fs layer
2022-09-27T15:26:51.0113154Z 4e5e6fbf81d0: Pulling fs layer
2022-09-27T15:26:51.0113523Z 1fd57c418b4f: Pulling fs layer
2022-09-27T15:26:51.0113895Z c252dd0cf292: Pulling fs layer
2022-09-27T15:26:51.0114273Z 6d1cf6420de5: Pulling fs layer
2022-09-27T15:26:51.0114666Z 22ff1eeb7cfc: Pulling fs layer
2022-09-27T15:26:51.0115037Z d8d2312a3e95: Pulling fs layer
2022-09-27T15:26:51.0115396Z 6b8fe070c229: Pulling fs layer
2022-09-27T15:26:51.0115705Z f860349178a3: Pulling fs layer
2022-09-27T15:26:51.0116002Z b9600c108e0a: Pulling fs layer
2022-09-27T15:26:51.0116316Z d7379e9f0242: Pulling fs layer
2022-09-27T15:26:51.0116631Z 111b3362d2ab: Pulling fs layer
2022-09-27T15:26:51.0116927Z 357e25b632e3: Pulling fs layer
2022-09-27T15:26:51.0117244Z 041ccbc6424a: Pulling fs layer
2022-09-27T15:26:51.0117548Z 6d1cf6420de5: Waiting
2022-09-27T15:26:51.0117856Z f83a7aa0ae33: Pulling fs layer
2022-09-27T15:26:51.0118188Z 7dd92e4c6366: Pulling fs layer
2022-09-27T15:26:51.0118503Z 22ff1eeb7cfc: Waiting
2022-09-27T15:26:51.0118782Z 4c3fc199e49b: Pulling fs layer
2022-09-27T15:26:51.0119089Z d8d2312a3e95: Waiting
2022-09-27T15:26:51.0119330Z 6b8fe070c229: Waiting
2022-09-27T15:26:51.0119546Z 1cf2d121678f: Pulling fs layer
2022-09-27T15:26:51.0119782Z 3d0511d3c79c: Pulling fs layer
2022-09-27T15:26:51.0120000Z f860349178a3: Waiting
2022-09-27T15:26:51.0120204Z 14b2bb0370e4: Pulling fs layer
2022-09-27T15:26:51.0120424Z b9600c108e0a: Waiting
2022-09-27T15:26:51.0120643Z 8da4048e870a: Pulling fs layer
2022-09-27T15:26:51.0120860Z df7557c6da37: Pulling fs layer
2022-09-27T15:26:51.0121242Z d7379e9f0242: Waiting
2022-09-27T15:26:51.0121464Z c47c4a7e8472: Pulling fs layer
2022-09-27T15:26:51.0121685Z fbbb9ed3e845: Pulling fs layer
2022-09-27T15:26:51.0121908Z 111b3362d2ab: Waiting
2022-09-27T15:26:51.0122128Z 25c87328e320: Pulling fs layer
2022-09-27T15:26:51.0122356Z 6ce6383c8b4c: Pulling fs layer
2022-09-27T15:26:51.0122629Z 805e81bf1b07: Pulling fs layer
2022-09-27T15:26:51.0122855Z 658a981e4c72: Pulling fs layer
2022-09-27T15:26:51.0123086Z 6e614c1c92e2: Pulling fs layer
2022-09-27T15:26:51.0123306Z 357e25b632e3: Waiting
2022-09-27T15:26:51.0123513Z ca8c741a00af: Pulling fs layer
2022-09-27T15:26:51.0123817Z ec20b9c0ab8c: Pulling fs layer
2022-09-27T15:26:51.0124043Z 041ccbc6424a: Waiting
2022-09-27T15:26:51.0124284Z 4778e25ffa15: Pulling fs layer
2022-09-27T15:26:51.0124506Z f83a7aa0ae33: Waiting
2022-09-27T15:26:51.0124727Z 6178951b0ba1: Pulling fs layer
2022-09-27T15:26:51.0124933Z 7dd92e4c6366: Waiting
2022-09-27T15:26:51.0125140Z 3d0511d3c79c: Waiting
2022-09-27T15:26:51.0125365Z 5f80c355afff: Pulling fs layer
2022-09-27T15:26:51.0125605Z 8fd17095407b: Pulling fs layer
2022-09-27T15:26:51.0125824Z 14b2bb0370e4: Waiting
2022-09-27T15:26:51.0126044Z 7d3e30f68342: Pulling fs layer
2022-09-27T15:26:51.0126258Z df7d91b50502: Pulling fs layer
2022-09-27T15:26:51.0126489Z ddc86fc3214e: Pulling fs layer
2022-09-27T15:26:51.0126710Z 4c3fc199e49b: Waiting
2022-09-27T15:26:51.0126915Z 46ee14c7b526: Pulling fs layer
2022-09-27T15:26:51.0127142Z 9ce9a24e805b: Pulling fs layer
2022-09-27T15:26:51.0127359Z 1cf2d121678f: Waiting
2022-09-27T15:26:51.0127565Z 19cd914bfdcb: Pulling fs layer
2022-09-27T15:26:51.0127800Z a7f4338ecba2: Pulling fs layer
2022-09-27T15:26:51.0128029Z c75978fac091: Pulling fs layer
2022-09-27T15:26:51.0128247Z f89a91cff893: Pulling fs layer
2022-09-27T15:26:51.0128474Z f345365236d1: Pulling fs layer
2022-09-27T15:26:51.0128701Z 4a3f3d395eb7: Pulling fs layer
2022-09-27T15:26:51.0128919Z e421c81bb4cf: Pulling fs layer
2022-09-27T15:26:51.0129144Z 775c0184544e: Pulling fs layer
2022-09-27T15:26:51.0129371Z 5a69da2170e4: Pulling fs layer
2022-09-27T15:26:51.0129586Z d3273d80d5f9: Pulling fs layer
2022-09-27T15:26:51.0129811Z f5905b45d58c: Pulling fs layer
2022-09-27T15:26:51.0130028Z dac0b0909394: Waiting
2022-09-27T15:26:51.0130232Z 42b9d02069e0: Pulling fs layer
2022-09-27T15:26:51.0130460Z c00b10f24e83: Pulling fs layer
2022-09-27T15:26:51.0130690Z 93c6c1ce6a86: Pulling fs layer
2022-09-27T15:26:51.0130916Z 239233a60227: Pulling fs layer
2022-09-27T15:26:51.0131120Z ddc86fc3214e: Waiting
2022-09-27T15:26:51.0131328Z 8da4048e870a: Waiting
2022-09-27T15:26:51.0131536Z ca8c741a00af: Waiting
2022-09-27T15:26:51.0131733Z 46ee14c7b526: Waiting
2022-09-27T15:26:51.0131943Z ec20b9c0ab8c: Waiting
2022-09-27T15:26:51.0132175Z df7557c6da37: Waiting
2022-09-27T15:26:51.0132369Z 9ce9a24e805b: Waiting
2022-09-27T15:26:51.0132577Z 4778e25ffa15: Waiting
2022-09-27T15:26:51.0132807Z 19cd914bfdcb: Waiting
2022-09-27T15:26:51.0132999Z c47c4a7e8472: Waiting
2022-09-27T15:26:51.0133203Z 6178951b0ba1: Waiting
2022-09-27T15:26:51.0133431Z a7f4338ecba2: Waiting
2022-09-27T15:26:51.0133624Z c75978fac091: Waiting
2022-09-27T15:26:51.0133832Z fbbb9ed3e845: Waiting
2022-09-27T15:26:51.0134042Z 5f80c355afff: Waiting
2022-09-27T15:26:51.0134236Z f89a91cff893: Waiting
2022-09-27T15:26:51.0134458Z f345365236d1: Waiting
2022-09-27T15:26:51.0134658Z 25c87328e320: Waiting
2022-09-27T15:26:51.0134847Z 1fd57c418b4f: Waiting
2022-09-27T15:26:51.0135053Z 8fd17095407b: Waiting
2022-09-27T15:26:51.0135259Z 4a3f3d395eb7: Waiting
2022-09-27T15:26:51.0135451Z 6ce6383c8b4c: Waiting
2022-09-27T15:26:51.0135653Z 7d3e30f68342: Waiting
2022-09-27T15:26:51.0135859Z 6e614c1c92e2: Waiting
2022-09-27T15:26:51.0136050Z df7d91b50502: Waiting
2022-09-27T15:26:51.0136256Z e421c81bb4cf: Waiting
2022-09-27T15:26:51.0136462Z 805e81bf1b07: Waiting
2022-09-27T15:26:51.0136653Z 775c0184544e: Waiting
2022-09-27T15:26:51.0136855Z c00b10f24e83: Waiting
2022-09-27T15:26:51.0137061Z 658a981e4c72: Waiting
2022-09-27T15:26:51.0137251Z 5a69da2170e4: Waiting
2022-09-27T15:26:51.0137519Z 93c6c1ce6a86: Waiting
2022-09-27T15:26:51.0137726Z c252dd0cf292: Waiting
2022-09-27T15:26:51.0137919Z d3273d80d5f9: Waiting
2022-09-27T15:26:51.0138126Z f5905b45d58c: Waiting
2022-09-27T15:26:51.0138332Z 42b9d02069e0: Waiting
2022-09-27T15:26:51.0138527Z 4e5e6fbf81d0: Waiting
2022-09-27T15:26:51.0847434Z 7249ff1ca2cc: Verifying Checksum
2022-09-27T15:26:51.0847795Z 7249ff1ca2cc: Download complete
2022-09-27T15:26:51.1656604Z dac0b0909394: Verifying Checksum
2022-09-27T15:26:51.1656883Z dac0b0909394: Download complete
2022-09-27T15:26:51.2535785Z 4e5e6fbf81d0: Verifying Checksum
2022-09-27T15:26:51.2536175Z 4e5e6fbf81d0: Download complete
2022-09-27T15:26:51.3396160Z 1fd57c418b4f: Download complete
2022-09-27T15:26:51.3528525Z 675920708c8b: Download complete
2022-09-27T15:26:51.4471895Z c252dd0cf292: Verifying Checksum
2022-09-27T15:26:51.4472641Z c252dd0cf292: Download complete
2022-09-27T15:26:51.4572099Z 6d1cf6420de5: Verifying Checksum
2022-09-27T15:26:51.4572387Z 6d1cf6420de5: Download complete
2022-09-27T15:26:51.5234907Z 22ff1eeb7cfc: Download complete
2022-09-27T15:26:51.5377533Z d8d2312a3e95: Verifying Checksum
2022-09-27T15:26:51.5377841Z d8d2312a3e95: Download complete
2022-09-27T15:26:51.6312385Z f860349178a3: Verifying Checksum
2022-09-27T15:26:51.6312768Z f860349178a3: Download complete
2022-09-27T15:26:51.7481486Z b9600c108e0a: Verifying Checksum
2022-09-27T15:26:51.7481743Z b9600c108e0a: Download complete
2022-09-27T15:26:52.0688420Z 675920708c8b: Pull complete
2022-09-27T15:26:52.1788012Z 7249ff1ca2cc: Pull complete
2022-09-27T15:26:52.4060904Z 6b8fe070c229: Verifying Checksum
2022-09-27T15:26:52.4061200Z 6b8fe070c229: Download complete
2022-09-27T15:26:52.4906678Z 111b3362d2ab: Verifying Checksum
2022-09-27T15:26:52.4907047Z 111b3362d2ab: Download complete
2022-09-27T15:26:52.5651204Z 357e25b632e3: Download complete
2022-09-27T15:26:53.0714856Z 041ccbc6424a: Verifying Checksum
2022-09-27T15:26:53.0715201Z 041ccbc6424a: Download complete
2022-09-27T15:26:53.1648752Z f83a7aa0ae33: Verifying Checksum
2022-09-27T15:26:53.1649143Z f83a7aa0ae33: Download complete
2022-09-27T15:26:53.2811242Z 7dd92e4c6366: Verifying Checksum
2022-09-27T15:26:53.2811654Z 7dd92e4c6366: Download complete
2022-09-27T15:26:53.3602647Z 4c3fc199e49b: Verifying Checksum
2022-09-27T15:26:53.3602983Z 4c3fc199e49b: Download complete
2022-09-27T15:26:53.4323143Z 1cf2d121678f: Download complete
2022-09-27T15:26:53.5087163Z 3d0511d3c79c: Verifying Checksum
2022-09-27T15:26:53.5087414Z 3d0511d3c79c: Download complete
2022-09-27T15:26:53.5867903Z 14b2bb0370e4: Verifying Checksum
2022-09-27T15:26:53.5868163Z 14b2bb0370e4: Download complete
2022-09-27T15:26:53.6915072Z 8da4048e870a: Download complete
2022-09-27T15:26:54.1291862Z a1a4ad211d43: Verifying Checksum
2022-09-27T15:26:54.1292131Z a1a4ad211d43: Download complete
2022-09-27T15:26:54.2052975Z c47c4a7e8472: Verifying Checksum
2022-09-27T15:26:54.2053458Z c47c4a7e8472: Download complete
2022-09-27T15:26:54.2887154Z fbbb9ed3e845: Download complete
2022-09-27T15:26:54.3746040Z 25c87328e320: Verifying Checksum
2022-09-27T15:26:54.3746441Z 25c87328e320: Download complete
2022-09-27T15:26:54.4447717Z 6ce6383c8b4c: Verifying Checksum
2022-09-27T15:26:54.4449570Z 6ce6383c8b4c: Download complete
2022-09-27T15:26:54.5165298Z 805e81bf1b07: Verifying Checksum
2022-09-27T15:26:54.5165669Z 805e81bf1b07: Download complete
2022-09-27T15:26:54.6416488Z df7557c6da37: Verifying Checksum
2022-09-27T15:26:54.6416791Z df7557c6da37: Download complete
2022-09-27T15:26:54.7169610Z 6e614c1c92e2: Verifying Checksum
2022-09-27T15:26:54.7169967Z 6e614c1c92e2: Download complete
2022-09-27T15:26:54.7913172Z ca8c741a00af: Download complete
2022-09-27T15:26:54.8699380Z ec20b9c0ab8c: Verifying Checksum
2022-09-27T15:26:54.8699846Z ec20b9c0ab8c: Download complete
2022-09-27T15:26:54.9525904Z 4778e25ffa15: Download complete
2022-09-27T15:26:55.0119419Z 6178951b0ba1: Download complete
2022-09-27T15:26:55.0979200Z 5f80c355afff: Download complete
2022-09-27T15:26:55.1550147Z 8fd17095407b: Verifying Checksum
2022-09-27T15:26:55.1550644Z 8fd17095407b: Download complete
2022-09-27T15:26:55.2987712Z df7d91b50502: Download complete
2022-09-27T15:26:55.3573477Z ddc86fc3214e: Verifying Checksum
2022-09-27T15:26:55.3573927Z ddc86fc3214e: Download complete
2022-09-27T15:26:55.4423236Z 46ee14c7b526: Verifying Checksum
2022-09-27T15:26:55.4423659Z 46ee14c7b526: Download complete
2022-09-27T15:26:55.5153448Z 9ce9a24e805b: Verifying Checksum
2022-09-27T15:26:55.5153897Z 9ce9a24e805b: Download complete
2022-09-27T15:26:55.9855759Z 19cd914bfdcb: Verifying Checksum
2022-09-27T15:26:55.9856978Z 19cd914bfdcb: Download complete
2022-09-27T15:26:56.0660030Z a7f4338ecba2: Verifying Checksum
2022-09-27T15:26:56.0660355Z a7f4338ecba2: Download complete
2022-09-27T15:26:56.1277343Z c75978fac091: Verifying Checksum
2022-09-27T15:26:56.1277634Z c75978fac091: Download complete
2022-09-27T15:26:56.1921141Z f89a91cff893: Verifying Checksum
2022-09-27T15:26:56.1923528Z f89a91cff893: Download complete
2022-09-27T15:26:56.2704648Z f345365236d1: Verifying Checksum
2022-09-27T15:26:56.2705008Z f345365236d1: Download complete
2022-09-27T15:26:56.5258970Z 4a3f3d395eb7: Verifying Checksum
2022-09-27T15:26:56.5259244Z 4a3f3d395eb7: Download complete
2022-09-27T15:26:56.5922958Z e421c81bb4cf: Download complete
2022-09-27T15:26:56.6566952Z 775c0184544e: Verifying Checksum
2022-09-27T15:26:56.6567910Z 775c0184544e: Download complete
2022-09-27T15:26:56.7612491Z 658a981e4c72: Verifying Checksum
2022-09-27T15:26:56.7612922Z 658a981e4c72: Download complete
2022-09-27T15:26:56.8304668Z d3273d80d5f9: Verifying Checksum
2022-09-27T15:26:56.8305095Z d3273d80d5f9: Download complete
2022-09-27T15:26:56.8911962Z f5905b45d58c: Download complete
2022-09-27T15:26:56.9674111Z 42b9d02069e0: Download complete
2022-09-27T15:26:57.0440167Z c00b10f24e83: Download complete
2022-09-27T15:26:57.1133356Z 93c6c1ce6a86: Verifying Checksum
2022-09-27T15:26:57.1133751Z 93c6c1ce6a86: Download complete
2022-09-27T15:26:57.7022515Z 239233a60227: Verifying Checksum
2022-09-27T15:26:57.7023638Z 239233a60227: Download complete
2022-09-27T15:26:59.4080294Z 5a69da2170e4: Verifying Checksum
2022-09-27T15:26:59.4080596Z 5a69da2170e4: Download complete
2022-09-27T15:27:00.7157385Z a1a4ad211d43: Pull complete
2022-09-27T15:27:00.8953729Z dac0b0909394: Pull complete
2022-09-27T15:27:01.0494177Z 4e5e6fbf81d0: Pull complete
2022-09-27T15:27:01.1858268Z 1fd57c418b4f: Pull complete
2022-09-27T15:27:01.3236921Z c252dd0cf292: Pull complete
2022-09-27T15:27:01.5502910Z 6d1cf6420de5: Pull complete
2022-09-27T15:27:01.7908841Z 22ff1eeb7cfc: Pull complete
2022-09-27T15:27:02.0381969Z d8d2312a3e95: Pull complete
2022-09-27T15:27:03.9036765Z 6b8fe070c229: Pull complete
2022-09-27T15:27:04.1392445Z f860349178a3: Pull complete
2022-09-27T15:27:04.3447391Z b9600c108e0a: Pull complete
2022-09-27T15:27:09.5931387Z d7379e9f0242: Verifying Checksum
2022-09-27T15:27:09.5931649Z d7379e9f0242: Download complete
2022-09-27T15:27:33.7532745Z d7379e9f0242: Pull complete
2022-09-27T15:27:34.1932401Z 111b3362d2ab: Pull complete
2022-09-27T15:27:34.2936904Z 357e25b632e3: Pull complete
2022-09-27T15:27:35.0682880Z 041ccbc6424a: Pull complete
2022-09-27T15:27:35.1692178Z f83a7aa0ae33: Pull complete
2022-09-27T15:27:35.2764078Z 7dd92e4c6366: Pull complete
2022-09-27T15:27:35.3498821Z 4c3fc199e49b: Pull complete
2022-09-27T15:27:35.4453376Z 1cf2d121678f: Pull complete
2022-09-27T15:27:35.5206941Z 3d0511d3c79c: Pull complete
2022-09-27T15:27:35.5970765Z 14b2bb0370e4: Pull complete
2022-09-27T15:27:35.6763265Z 8da4048e870a: Pull complete
2022-09-27T15:27:37.5294610Z df7557c6da37: Pull complete
2022-09-27T15:27:37.6764408Z c47c4a7e8472: Pull complete
2022-09-27T15:27:37.9019809Z fbbb9ed3e845: Pull complete
2022-09-27T15:27:38.2069143Z 25c87328e320: Pull complete
2022-09-27T15:27:38.4429238Z 6ce6383c8b4c: Pull complete
2022-09-27T15:27:38.6616671Z 805e81bf1b07: Pull complete
2022-09-27T15:27:43.2099898Z 658a981e4c72: Pull complete
2022-09-27T15:27:43.4249513Z 6e614c1c92e2: Pull complete
2022-09-27T15:27:43.6459790Z ca8c741a00af: Pull complete
2022-09-27T15:27:43.8938863Z ec20b9c0ab8c: Pull complete
2022-09-27T15:27:44.1333906Z 4778e25ffa15: Pull complete
2022-09-27T15:27:44.3516267Z 6178951b0ba1: Pull complete
2022-09-27T15:27:44.5069359Z 5f80c355afff: Pull complete
2022-09-27T15:27:44.5972106Z 8fd17095407b: Pull complete
2022-09-27T15:27:44.7054033Z 7d3e30f68342: Pull complete
2022-09-27T15:27:44.7870826Z df7d91b50502: Pull complete
2022-09-27T15:27:44.8856938Z ddc86fc3214e: Pull complete
2022-09-27T15:27:44.9768418Z 46ee14c7b526: Pull complete
2022-09-27T15:27:45.0741882Z 9ce9a24e805b: Pull complete
2022-09-27T15:27:46.1517718Z 19cd914bfdcb: Pull complete
2022-09-27T15:27:46.2287532Z a7f4338ecba2: Pull complete
2022-09-27T15:27:46.3128192Z c75978fac091: Pull complete
2022-09-27T15:27:46.4155729Z f89a91cff893: Pull complete
2022-09-27T15:27:46.5052588Z f345365236d1: Pull complete
2022-09-27T15:27:46.7861758Z 4a3f3d395eb7: Pull complete
2022-09-27T15:27:46.8656088Z e421c81bb4cf: Pull complete
2022-09-27T15:27:46.9689814Z 775c0184544e: Pull complete
2022-09-27T15:27:50.6620687Z 5a69da2170e4: Pull complete
2022-09-27T15:27:50.8406226Z d3273d80d5f9: Pull complete
2022-09-27T15:27:50.9287162Z f5905b45d58c: Pull complete
2022-09-27T15:27:51.0192726Z 42b9d02069e0: Pull complete
2022-09-27T15:27:51.1254053Z c00b10f24e83: Pull complete
2022-09-27T15:27:51.2096378Z 93c6c1ce6a86: Pull complete
2022-09-27T15:27:52.6470796Z 239233a60227: Pull complete
2022-09-27T15:27:52.6561729Z Digest: sha256:aa31a48de2d62af84cacd725abaa1599e63c6bb3385fa6e176b6d91f90a39348
2022-09-27T15:27:52.6595805Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:27:52.6619108Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:27:52.6676852Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-09-27T15:27:52.6677133Z python3 -m pip install psutil==5.9.1
2022-09-27T15:27:52.6677387Z python3 -m pip install pynvml==11.4.1
2022-09-27T15:27:52.6677647Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-09-27T15:27:52.6677917Z echo "::set-output name=monitor-script-pid::${!}"
2022-09-27T15:27:52.6688847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:27:52.6689066Z env:
2022-09-27T15:27:52.6689227Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:27:52.6689416Z ##[endgroup]
2022-09-27T15:27:53.2421518Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:27:53.2633698Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-09-27T15:27:53.6630849Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:27:53.6806280Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-09-27T15:27:53.8596102Z Prepare all required actions
2022-09-27T15:27:53.8596391Z Getting action download info
2022-09-27T15:27:54.0444882Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5)
2022-09-27T15:27:54.2206338Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-09-27T15:27:54.3450456Z ##[group]Run ./.github/actions/download-build-artifacts
2022-09-27T15:27:54.3450843Z with:
2022-09-27T15:27:54.3451128Z   name: linux-focal-py3.7-gcc7
2022-09-27T15:27:54.3451436Z env:
2022-09-27T15:27:54.3451729Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:27:54.3452023Z ##[endgroup]
2022-09-27T15:27:54.3482606Z ##[group]Run seemethere/download-artifact-s3@v4
2022-09-27T15:27:54.3482823Z with:
2022-09-27T15:27:54.3483010Z   name: linux-focal-py3.7-gcc7
2022-09-27T15:27:54.3483208Z   s3-bucket: gha-artifacts
2022-09-27T15:27:54.3483406Z   region: us-east-1
2022-09-27T15:27:54.3483576Z env:
2022-09-27T15:27:54.3483772Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:27:54.3484034Z ##[endgroup]
2022-09-27T15:27:54.7285404Z Found 1 objects with prefix pytorch/pytorch/3133193930/linux-focal-py3.7-gcc7/
2022-09-27T15:27:54.7285890Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:27:57.7380113Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:27:57.7380456Z 
2022-09-27T15:27:57.7381605Z Artifact download has finished successfully
2022-09-27T15:27:57.7515931Z ##[group]Run unzip -o artifacts.zip
2022-09-27T15:27:57.7516157Z unzip -o artifacts.zip
2022-09-27T15:27:57.7527217Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:27:57.7527425Z env:
2022-09-27T15:27:57.7527592Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:27:57.7527772Z ##[endgroup]
2022-09-27T15:27:57.7828840Z Archive:  artifacts.zip
2022-09-27T15:27:57.7829850Z    creating: dist/
2022-09-27T15:27:58.5353716Z   inflating: dist/torch-1.13.0a0+git52424e2-cp37-cp37m-linux_x86_64.whl  
2022-09-27T15:27:58.5354056Z    creating: build/custom_test_artifacts/
2022-09-27T15:27:58.5354358Z    creating: build/custom_test_artifacts/custom-op-build/
2022-09-27T15:27:58.5354717Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-09-27T15:27:58.5356531Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:27:58.5356940Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-09-27T15:27:58.5357355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:27:58.5357784Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:27:58.5358208Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:27:58.5359878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:27:58.5361427Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:27:58.5361926Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:27:58.5362354Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:27:58.5363720Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:27:58.5364999Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:27:58.5366349Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:27:58.5366931Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:27:58.5368284Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:27:58.5368958Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:27:58.5369545Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-09-27T15:27:58.5370100Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-09-27T15:27:58.5370706Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:27:58.5371268Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-09-27T15:27:58.5371929Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-09-27T15:27:58.5372454Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-09-27T15:27:58.5373100Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-09-27T15:27:58.5373710Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-09-27T15:27:58.5374292Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:27:58.5374744Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-09-27T15:27:58.5375182Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-09-27T15:27:58.5375630Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-09-27T15:27:58.5376074Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-09-27T15:27:58.5389753Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-09-27T15:27:58.5476560Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-09-27T15:27:58.5477175Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-09-27T15:27:58.5477768Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-09-27T15:27:58.5478458Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-09-27T15:27:58.5478997Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-09-27T15:27:58.5479604Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-09-27T15:27:58.5480256Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:27:58.5480851Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-09-27T15:27:58.5481303Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-09-27T15:27:58.5481763Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-09-27T15:27:58.5482222Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-09-27T15:27:58.5496308Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-09-27T15:27:58.5558131Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-09-27T15:27:58.5558763Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:27:58.5559297Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:27:58.5559836Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-09-27T15:27:58.5560331Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-09-27T15:27:58.5560858Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:27:58.5562029Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-09-27T15:27:58.5562955Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-09-27T15:27:58.5563388Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-09-27T15:27:58.5633960Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-09-27T15:27:58.5681260Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-09-27T15:27:58.5681611Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-09-27T15:27:58.5681955Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-09-27T15:27:58.5684268Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:27:58.5684687Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-09-27T15:27:58.5685241Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:27:58.5685659Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:27:58.5686082Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:27:58.5687409Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:27:58.5688695Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:27:58.5689129Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:27:58.5689550Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:27:58.5691192Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:27:58.5692477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:27:58.5693781Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:27:58.5694242Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:27:58.5695621Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:27:58.5696346Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:27:58.5696757Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-09-27T15:27:58.5697153Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-09-27T15:27:58.5697559Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:27:58.5697958Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-09-27T15:27:58.5698405Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-09-27T15:27:58.5698877Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-09-27T15:27:58.5699328Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-09-27T15:27:58.5699888Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-09-27T15:27:58.5700334Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-09-27T15:27:58.5700780Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-09-27T15:27:58.5701321Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-09-27T15:27:58.5701857Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-09-27T15:27:58.5702315Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-09-27T15:27:58.5717796Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-09-27T15:27:58.5766160Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-09-27T15:27:58.5766641Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:27:58.5767089Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:27:58.5767511Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-09-27T15:27:58.5767916Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-09-27T15:27:58.5768489Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:27:58.5769858Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-09-27T15:27:58.5770375Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-09-27T15:27:58.5770864Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-09-27T15:27:58.5808430Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-09-27T15:27:58.5808803Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-09-27T15:27:58.5809173Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-09-27T15:27:58.5811714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:27:58.5812234Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-09-27T15:27:58.5812683Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:27:58.5813123Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:27:58.5813570Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:27:58.5814515Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:27:58.5815934Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:27:58.5816376Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:27:58.5816818Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:27:58.5818391Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:27:58.5819672Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:27:58.5821004Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:27:58.5821493Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:27:58.5822998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:27:58.5823549Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:27:58.5824185Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-09-27T15:27:58.5824612Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-09-27T15:27:58.5825053Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:27:58.5825569Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-09-27T15:27:58.5826120Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-09-27T15:27:58.5826616Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-09-27T15:27:58.5827106Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-09-27T15:27:58.5827561Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-09-27T15:27:58.5828040Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:27:58.5828517Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-09-27T15:27:58.5828999Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-09-27T15:27:58.5829462Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-09-27T15:27:58.5829999Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-09-27T15:27:58.5832372Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-09-27T15:27:58.5944812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-09-27T15:27:58.5945295Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-09-27T15:27:58.5945848Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-09-27T15:27:58.5946362Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-09-27T15:27:58.5946858Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-09-27T15:27:58.5947328Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-09-27T15:27:58.5947812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:27:58.5948304Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-09-27T15:27:58.5948792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-09-27T15:27:58.5949264Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-09-27T15:27:58.5949751Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-09-27T15:27:58.5964649Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-09-27T15:27:58.6008661Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-09-27T15:27:58.6009167Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:27:58.6009646Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:27:58.6010090Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-09-27T15:27:58.6010512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-09-27T15:27:58.6011173Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:27:58.6012665Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-09-27T15:27:58.6013361Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-09-27T15:27:58.6014069Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-09-27T15:27:58.6104819Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-09-27T15:27:58.6139436Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-09-27T15:27:58.6139714Z    creating: build/lib/
2022-09-27T15:27:58.6140293Z   inflating: build/lib/libclog.a     
2022-09-27T15:27:58.6191431Z   inflating: build/lib/libgtest.a    
2022-09-27T15:27:58.6199367Z   inflating: build/lib/libpthreadpool.a  
2022-09-27T15:27:58.6270349Z   inflating: build/lib/libbenchmark.a  
2022-09-27T15:27:58.6352456Z   inflating: build/lib/libprotobuf-lite.a  
2022-09-27T15:27:58.6359339Z   inflating: build/lib/libittnotify.a  
2022-09-27T15:27:58.6384096Z   inflating: build/lib/libtensorpipe_uv.a  
2022-09-27T15:27:58.6443839Z   inflating: build/lib/libasmjit.a   
2022-09-27T15:27:58.6533820Z   inflating: build/lib/libgloo.a     
2022-09-27T15:27:58.6948080Z   inflating: build/lib/libprotobuf.a  
2022-09-27T15:27:58.6963054Z   inflating: build/lib/libfmt.a      
2022-09-27T15:27:58.6963746Z   inflating: build/lib/libfoxi_loader.a  
2022-09-27T15:27:58.7019603Z   inflating: build/lib/libc10.so     
2022-09-27T15:27:58.7020662Z   inflating: build/lib/libtorch_global_deps.so  
2022-09-27T15:27:58.7028121Z   inflating: build/lib/libcpuinfo.a  
2022-09-27T15:27:58.7035151Z   inflating: build/lib/libcpuinfo_internals.a  
2022-09-27T15:27:58.7047314Z   inflating: build/lib/libqnnpack.a  
2022-09-27T15:27:58.7065971Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-09-27T15:27:58.7067851Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-09-27T15:27:58.7510676Z   inflating: build/lib/libprotoc.a   
2022-09-27T15:27:58.7525676Z   inflating: build/lib/libgmock.a    
2022-09-27T15:27:58.7526317Z   inflating: build/lib/libgtest_main.a  
2022-09-27T15:27:58.7526805Z   inflating: build/lib/libbenchmark_main.a  
2022-09-27T15:27:59.3793075Z   inflating: build/lib/libdnnl.a     
2022-09-27T15:27:59.3809877Z   inflating: build/lib/libnnpack.a   
2022-09-27T15:27:59.4315800Z   inflating: build/lib/libtensorpipe.a  
2022-09-27T15:27:59.4316713Z   inflating: build/lib/libgmock_main.a  
2022-09-27T15:27:59.5491783Z   inflating: build/lib/libfbgemm.a   
2022-09-27T15:27:59.6363266Z   inflating: build/lib/libdnnl_graph.a  
2022-09-27T15:27:59.6553521Z   inflating: build/lib/libkineto.a   
2022-09-27T15:27:59.6588532Z   inflating: build/lib/libcaffe2_protos.a  
2022-09-27T15:27:59.6625359Z   inflating: build/lib/libonnx_proto.a  
2022-09-27T15:27:59.6735250Z   inflating: build/lib/libXNNPACK.a  
2022-09-27T15:27:59.7260264Z   inflating: build/lib/libonnx.a     
2022-09-27T15:28:01.5095604Z   inflating: build/lib/libtorch_cpu.so  
2022-09-27T15:28:01.5096256Z   inflating: build/lib/libtorch.so   
2022-09-27T15:28:01.5115186Z   inflating: build/lib/libjitbackend_test.so  
2022-09-27T15:28:01.5161860Z   inflating: build/lib/libtorchbind_test.so  
2022-09-27T15:28:01.5185929Z   inflating: build/lib/libbackend_with_compiler.so  
2022-09-27T15:28:01.5189533Z   inflating: build/lib/libshm.so     
2022-09-27T15:28:01.6467217Z   inflating: build/lib/libtorch_python.so  
2022-09-27T15:28:01.6497982Z   inflating: build/lib/libnnapi_backend.so  
2022-09-27T15:28:01.6498532Z    creating: build/bin/
2022-09-27T15:28:01.6539604Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-09-27T15:28:01.6582789Z   inflating: build/bin/c10_DeviceGuard_test  
2022-09-27T15:28:01.6624925Z   inflating: build/bin/c10_Device_test  
2022-09-27T15:28:01.6674196Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-09-27T15:28:01.6714232Z   inflating: build/bin/c10_StreamGuard_test  
2022-09-27T15:28:01.6756160Z   inflating: build/bin/c10_SymInt_test  
2022-09-27T15:28:01.6802612Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-09-27T15:28:01.6849281Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-09-27T15:28:01.6897167Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-09-27T15:28:01.6937189Z   inflating: build/bin/c10_Array_test  
2022-09-27T15:28:01.6981476Z   inflating: build/bin/c10_Bitset_test  
2022-09-27T15:28:01.7021865Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-09-27T15:28:01.7064606Z   inflating: build/bin/c10_C++17_test  
2022-09-27T15:28:01.7105570Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-09-27T15:28:01.7152899Z   inflating: build/bin/c10_LeftRight_test  
2022-09-27T15:28:01.7194497Z   inflating: build/bin/c10_Half_test  
2022-09-27T15:28:01.7246936Z   inflating: build/bin/c10_Metaprogramming_test  
2022-09-27T15:28:01.7288670Z   inflating: build/bin/c10_Synchronized_test  
2022-09-27T15:28:01.7336374Z   inflating: build/bin/c10_ThreadLocal_test  
2022-09-27T15:28:01.7457090Z   inflating: build/bin/c10_SmallVectorTest  
2022-09-27T15:28:01.7500972Z   inflating: build/bin/c10_TypeIndex_test  
2022-09-27T15:28:01.7543052Z   inflating: build/bin/c10_TypeList_test  
2022-09-27T15:28:01.7583164Z   inflating: build/bin/c10_TypeTraits_test  
2022-09-27T15:28:01.7626467Z   inflating: build/bin/c10_accumulate_test  
2022-09-27T15:28:01.7673314Z   inflating: build/bin/c10_bfloat16_test  
2022-09-27T15:28:01.7718219Z   inflating: build/bin/c10_complex_math_test  
2022-09-27T15:28:01.7764798Z   inflating: build/bin/c10_complex_test  
2022-09-27T15:28:01.7856198Z   inflating: build/bin/c10_either_test  
2022-09-27T15:28:01.7899726Z   inflating: build/bin/c10_exception_test  
2022-09-27T15:28:01.7941656Z   inflating: build/bin/c10_flags_test  
2022-09-27T15:28:01.7983705Z   inflating: build/bin/c10_irange_test  
2022-09-27T15:28:01.8124764Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-09-27T15:28:01.8173043Z   inflating: build/bin/c10_logging_test  
2022-09-27T15:28:01.8235895Z   inflating: build/bin/c10_optional_test  
2022-09-27T15:28:01.8287812Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-09-27T15:28:01.8333539Z   inflating: build/bin/c10_registry_test  
2022-09-27T15:28:01.8382882Z   inflating: build/bin/c10_string_view_test  
2022-09-27T15:28:01.8425618Z   inflating: build/bin/c10_tempfile_test  
2022-09-27T15:28:01.8472973Z   inflating: build/bin/c10_typeid_test  
2022-09-27T15:28:01.8520244Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-09-27T15:28:01.8929768Z   inflating: build/bin/protoc-3.13.0.0  
2022-09-27T15:28:01.9341366Z   inflating: build/bin/protoc        
2022-09-27T15:28:01.9617091Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-09-27T15:28:01.9863766Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-09-27T15:28:01.9909049Z   inflating: build/bin/FileStoreTest  
2022-09-27T15:28:01.9954060Z   inflating: build/bin/HashStoreTest  
2022-09-27T15:28:02.0004723Z   inflating: build/bin/TCPStoreTest  
2022-09-27T15:28:02.0007310Z   inflating: build/bin/example_allreduce  
2022-09-27T15:28:02.0061957Z   inflating: build/bin/ProcessGroupGlooTest  
2022-09-27T15:28:02.0106496Z   inflating: build/bin/Dimname_test  
2022-09-27T15:28:02.0167440Z   inflating: build/bin/Dict_test     
2022-09-27T15:28:02.0220888Z   inflating: build/bin/MaybeOwned_test  
2022-09-27T15:28:02.0269148Z   inflating: build/bin/NamedTensor_test  
2022-09-27T15:28:02.0318683Z   inflating: build/bin/apply_utils_test  
2022-09-27T15:28:02.0369351Z   inflating: build/bin/basic         
2022-09-27T15:28:02.0418668Z   inflating: build/bin/atest         
2022-09-27T15:28:02.0463792Z   inflating: build/bin/broadcast_test  
2022-09-27T15:28:02.0513132Z   inflating: build/bin/cpu_generator_test  
2022-09-27T15:28:02.0557400Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-09-27T15:28:02.0599149Z   inflating: build/bin/dispatch_key_set_test  
2022-09-27T15:28:02.0640614Z   inflating: build/bin/dlconvertor_test  
2022-09-27T15:28:02.0714520Z   inflating: build/bin/cpu_rng_test  
2022-09-27T15:28:02.0763616Z   inflating: build/bin/extension_backend_test  
2022-09-27T15:28:02.0810157Z   inflating: build/bin/half_test     
2022-09-27T15:28:02.0889071Z   inflating: build/bin/ivalue_test   
2022-09-27T15:28:02.0930488Z   inflating: build/bin/lazy_tensor_test  
2022-09-27T15:28:02.0974391Z   inflating: build/bin/math_kernel_test  
2022-09-27T15:28:02.1018284Z   inflating: build/bin/memory_format_test  
2022-09-27T15:28:02.1062432Z   inflating: build/bin/memory_overlapping_test  
2022-09-27T15:28:02.1105125Z   inflating: build/bin/operator_name_test  
2022-09-27T15:28:02.1152059Z   inflating: build/bin/native_test   
2022-09-27T15:28:02.1195774Z   inflating: build/bin/mobile_memory_cleanup  
2022-09-27T15:28:02.1237654Z   inflating: build/bin/operators_test  
2022-09-27T15:28:02.1281645Z   inflating: build/bin/packedtensoraccessor_test  
2022-09-27T15:28:02.1329742Z   inflating: build/bin/quantized_test  
2022-09-27T15:28:02.1384175Z   inflating: build/bin/pow_test      
2022-09-27T15:28:02.1426806Z   inflating: build/bin/reportMemoryUsage_test  
2022-09-27T15:28:02.1468198Z   inflating: build/bin/reduce_ops_test  
2022-09-27T15:28:02.1515647Z   inflating: build/bin/scalar_tensor_test  
2022-09-27T15:28:02.1563677Z   inflating: build/bin/scalar_test   
2022-09-27T15:28:02.1606723Z   inflating: build/bin/stride_properties_test  
2022-09-27T15:28:02.1672123Z   inflating: build/bin/tensor_iterator_test  
2022-09-27T15:28:02.1718774Z   inflating: build/bin/type_ptr_test  
2022-09-27T15:28:02.1721354Z   inflating: build/bin/thread_init_test  
2022-09-27T15:28:02.1762842Z   inflating: build/bin/variant_test  
2022-09-27T15:28:02.1809437Z   inflating: build/bin/test_parallel  
2022-09-27T15:28:02.1853017Z   inflating: build/bin/undefined_tensor_test  
2022-09-27T15:28:02.1903764Z   inflating: build/bin/type_test     
2022-09-27T15:28:02.1905282Z   inflating: build/bin/verify_api_visibility  
2022-09-27T15:28:02.1963333Z   inflating: build/bin/vmap_test     
2022-09-27T15:28:02.2005938Z   inflating: build/bin/weakref_test  
2022-09-27T15:28:02.2048312Z   inflating: build/bin/wrapdim_test  
2022-09-27T15:28:02.2140159Z   inflating: build/bin/List_test     
2022-09-27T15:28:02.2180986Z   inflating: build/bin/xla_tensor_test  
2022-09-27T15:28:02.2231333Z   inflating: build/bin/IListRef_test  
2022-09-27T15:28:02.2334150Z   inflating: build/bin/kernel_function_legacy_test  
2022-09-27T15:28:02.2415269Z   inflating: build/bin/kernel_function_test  
2022-09-27T15:28:02.2469533Z   inflating: build/bin/KernelFunction_test  
2022-09-27T15:28:02.2577847Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-09-27T15:28:02.2665604Z   inflating: build/bin/kernel_lambda_test  
2022-09-27T15:28:02.2716044Z   inflating: build/bin/kernel_stackbased_test  
2022-09-27T15:28:02.2758219Z   inflating: build/bin/CppSignature_test  
2022-09-27T15:28:02.2839228Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-09-27T15:28:02.2879340Z   inflating: build/bin/op_allowlist_test  
2022-09-27T15:28:02.2923855Z   inflating: build/bin/inline_container_test  
2022-09-27T15:28:02.3167448Z   inflating: build/bin/op_registration_test  
2022-09-27T15:28:02.3214658Z   inflating: build/bin/backend_fallback_test  
2022-09-27T15:28:02.3228251Z   inflating: build/bin/tutorial_tensorexpr  
2022-09-27T15:28:02.3273230Z   inflating: build/bin/test_dist_autograd  
2022-09-27T15:28:02.3275362Z   inflating: build/bin/parallel_benchmark  
2022-09-27T15:28:02.3333764Z   inflating: build/bin/test_cpp_rpc  
2022-09-27T15:28:02.3392033Z   inflating: build/bin/test_mobile_nnc  
2022-09-27T15:28:02.3401289Z   inflating: build/bin/aot_model_compiler_test  
2022-09-27T15:28:02.3694226Z   inflating: build/bin/test_lazy     
2022-09-27T15:28:02.4372075Z   inflating: build/bin/test_tensorexpr  
2022-09-27T15:28:02.4376453Z   inflating: build/bin/torch_shm_manager  
2022-09-27T15:28:02.5379634Z   inflating: build/bin/test_api      
2022-09-27T15:28:02.5870213Z   inflating: build/bin/test_jit      
2022-09-27T15:28:02.5871525Z   inflating: .pytorch-test-times.json  
2022-09-27T15:28:02.5894889Z ##[group]Run df -H
2022-09-27T15:28:02.5895069Z df -H
2022-09-27T15:28:02.5906503Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:28:02.5906801Z env:
2022-09-27T15:28:02.5906983Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:28:02.5907164Z ##[endgroup]
2022-09-27T15:28:02.5952154Z Filesystem      Size  Used Avail Use% Mounted on
2022-09-27T15:28:02.5952612Z devtmpfs        8.2G     0  8.2G   0% /dev
2022-09-27T15:28:02.5953005Z tmpfs           8.2G  447k  8.2G   1% /dev/shm
2022-09-27T15:28:02.5953210Z tmpfs           8.2G  431k  8.2G   1% /run
2022-09-27T15:28:02.5953425Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2022-09-27T15:28:02.5953664Z /dev/nvme0n1p1  162G   16G  146G  10% /
2022-09-27T15:28:02.5971294Z ##[group]Run .github/scripts/parse_ref.py
2022-09-27T15:28:02.5971539Z .github/scripts/parse_ref.py
2022-09-27T15:28:02.5981548Z shell: /usr/bin/bash -e {0}
2022-09-27T15:28:02.5981721Z env:
2022-09-27T15:28:02.5981897Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:28:02.5982084Z ##[endgroup]
2022-09-27T15:28:02.6201963Z ##[group]Run set -x
2022-09-27T15:28:02.6202232Z set -x
2022-09-27T15:28:02.6202387Z 
2022-09-27T15:28:02.6202586Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-09-27T15:28:02.6202931Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-09-27T15:28:02.6203174Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-09-27T15:28:02.6203414Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-09-27T15:28:02.6203618Z else
2022-09-27T15:28:02.6203809Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:28:02.6204005Z fi
2022-09-27T15:28:02.6204163Z 
2022-09-27T15:28:02.6204385Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-09-27T15:28:02.6204618Z 
2022-09-27T15:28:02.6204860Z # sanitize the input commit message and PR body here:
2022-09-27T15:28:02.6205133Z #
2022-09-27T15:28:02.6205402Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-09-27T15:28:02.6205779Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-09-27T15:28:02.6206084Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-09-27T15:28:02.6206317Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-09-27T15:28:02.6206504Z 
2022-09-27T15:28:02.6206751Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-09-27T15:28:02.6207029Z # wreak havoc internally
2022-09-27T15:28:02.6207268Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-09-27T15:28:02.6207511Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-09-27T15:28:02.6207691Z 
2022-09-27T15:28:02.6207915Z # detached container should get cleaned up by teardown_ec2_linux
2022-09-27T15:28:02.6208212Z # TODO: Stop building test binaries as part of the build phase
2022-09-27T15:28:02.6208473Z # Used for GPU_FLAG since that doesn't play nice
2022-09-27T15:28:02.6208712Z # shellcheck disable=SC2086,SC2090
2022-09-27T15:28:02.6208935Z container_name=$(docker run \
2022-09-27T15:28:02.6209137Z   ${GPU_FLAG:-} \
2022-09-27T15:28:02.6209321Z   -e BUILD_ENVIRONMENT \
2022-09-27T15:28:02.6209523Z   -e PR_NUMBER \
2022-09-27T15:28:02.6209720Z   -e GITHUB_ACTIONS \
2022-09-27T15:28:02.6209901Z   -e BASE_SHA \
2022-09-27T15:28:02.6210080Z   -e BRANCH \
2022-09-27T15:28:02.6210255Z   -e SHA1 \
2022-09-27T15:28:02.6210430Z   -e AWS_DEFAULT_REGION \
2022-09-27T15:28:02.6210629Z   -e IN_WHEEL_TEST \
2022-09-27T15:28:02.6210817Z   -e SHARD_NUMBER \
2022-09-27T15:28:02.6210993Z   -e TEST_CONFIG \
2022-09-27T15:28:02.6211184Z   -e NUM_TEST_SHARDS \
2022-09-27T15:28:02.6211377Z   -e PR_BODY \
2022-09-27T15:28:02.6211557Z   -e COMMIT_MESSAGES \
2022-09-27T15:28:02.6211767Z   -e PYTORCH_RETRY_TEST_CASES \
2022-09-27T15:28:02.6211996Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-09-27T15:28:02.6212194Z   -e PR_LABELS \
2022-09-27T15:28:02.6212400Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-09-27T15:28:02.6212612Z   -e SCCACHE_BUCKET \
2022-09-27T15:28:02.6212811Z   -e SCCACHE_S3_KEY_PREFIX \
2022-09-27T15:28:02.6212994Z   -e XLA_CUDA \
2022-09-27T15:28:02.6213201Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-09-27T15:28:02.6213449Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-09-27T15:28:02.6213672Z   --ulimit stack=10485760:83886080 \
2022-09-27T15:28:02.6213898Z   --security-opt seccomp=unconfined \
2022-09-27T15:28:02.6214121Z   --cap-add=SYS_PTRACE \
2022-09-27T15:28:02.6214301Z   --ipc=host \
2022-09-27T15:28:02.6214559Z   --shm-size="${SHM_SIZE}" \
2022-09-27T15:28:02.6214750Z   --tty \
2022-09-27T15:28:02.6214910Z   --detach \
2022-09-27T15:28:02.6215106Z   --name="${container_name}" \
2022-09-27T15:28:02.6215351Z   --user jenkins \
2022-09-27T15:28:02.6215584Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-09-27T15:28:02.6215823Z   -w /var/lib/jenkins/workspace \
2022-09-27T15:28:02.6216026Z   "${DOCKER_IMAGE}"
2022-09-27T15:28:02.6216200Z )
2022-09-27T15:28:02.6216433Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-09-27T15:28:02.6226721Z shell: /usr/bin/bash -e {0}
2022-09-27T15:28:02.6226908Z env:
2022-09-27T15:28:02.6227071Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:28:02.6227304Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-09-27T15:28:02.6227519Z   PR_NUMBER: 85462
2022-09-27T15:28:02.6227684Z   BRANCH: pull/85462
2022-09-27T15:28:02.6227898Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:28:02.6228137Z   BASE_SHA: 76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:28:02.6228344Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T15:28:02.6228558Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T15:28:02.6228764Z   TEST_CONFIG: distributed
2022-09-27T15:28:02.6228949Z   SHARD_NUMBER: 2
2022-09-27T15:28:02.6229107Z   NUM_TEST_SHARDS: 2
2022-09-27T15:28:02.6230814Z   PR_BODY: Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.

I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. 

Here is the code to reproduce the bug:

```
import torch
import weakref
from transformers import DetrForObjectDetection

class X:
    def __init__(self, optimizer):
        self.optimizer = optimizer

        # Will cause cyclic reference.
        self.func = self.dummy

        # Will work as expected, memory cleared after instance count is zero.
        # self.func = weakref.WeakMethod(self.dummy)

    def dummy(self, x):
        return 1.

def test():
    model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50')
    model.to('cuda')
    optimizer = torch.optim.Adam(model.parameters())
    x = X(optimizer)

test()
print(f'{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).
```
2022-09-27T15:28:02.6232698Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-09-27T15:28:02.6232928Z   SCCACHE_S3_KEY_PREFIX: pull
2022-09-27T15:28:02.6233117Z   SHM_SIZE: 1g
2022-09-27T15:28:02.6233456Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:28:02.6233775Z   XLA_CUDA: 
2022-09-27T15:28:02.6234035Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:28:02.6234295Z ##[endgroup]
2022-09-27T15:28:02.6257366Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-09-27T15:28:02.6257943Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]]
2022-09-27T15:28:02.6258264Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:28:02.6260140Z ++ git cherry -v origin/master
2022-09-27T15:28:02.6454601Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR
2022-09-27T15:28:02.6455450Z + acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR
2022-09-27T15:28:02.6455992Z + 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form
2022-09-27T15:28:02.6456618Z + 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference
2022-09-27T15:28:02.6457119Z + 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:28:02.6458982Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:28:02.6475065Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained('\''facebook/detr-resnet-50'\'')    model.to('\''cuda'\'')    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f'\''{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}'\'')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:28:02.6479665Z + export 'COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:28:02.6481752Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:28:02.6489826Z + export 'PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:28:02.6494170Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:28:02.6495958Z +++ nproc --ignore=2
2022-09-27T15:28:02.6531916Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3133193930 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:28:13.4068722Z + container_name=89d4c2f4be4530632f6fcecdaeba7f9c867a1bfa7fb3dd5b99f5b3cc80af06b2
2022-09-27T15:28:13.4069777Z + docker exec -t 89d4c2f4be4530632f6fcecdaeba7f9c867a1bfa7fb3dd5b99f5b3cc80af06b2 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-09-27T15:28:13.8306979Z Processing ./dist/torch-1.13.0a0+git52424e2-cp37-cp37m-linux_x86_64.whl
2022-09-27T15:28:13.9042987Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+git52424e2) (4.3.0)
2022-09-27T15:28:14.6842528Z Installing collected packages: torch
2022-09-27T15:28:20.8592522Z Successfully installed torch-1.13.0a0+git52424e2
2022-09-27T15:28:20.9906866Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-09-27T15:28:21.0061577Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2022-09-27T15:28:21.0062266Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2022-09-27T15:28:21.0062679Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2022-09-27T15:28:21.0063243Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2022-09-27T15:28:21.0063645Z + BUILD_DIR=build
2022-09-27T15:28:21.0064166Z + BUILD_RENAMED_DIR=build_renamed
2022-09-27T15:28:21.0064468Z + BUILD_BIN_DIR=build/bin
2022-09-27T15:28:21.0064775Z + export VALGRIND=ON
2022-09-27T15:28:21.0065042Z + VALGRIND=ON
2022-09-27T15:28:21.0065414Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]]
2022-09-27T15:28:21.0065976Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]]
2022-09-27T15:28:21.0066376Z ++ realpath build/custom_test_artifacts
2022-09-27T15:28:21.0072808Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-09-27T15:28:21.0075592Z ++ dirname .jenkins/pytorch/test.sh
2022-09-27T15:28:21.0081079Z + source .jenkins/pytorch/common.sh
2022-09-27T15:28:21.0084450Z +++ dirname .jenkins/pytorch/common.sh
2022-09-27T15:28:21.0089645Z ++ source .jenkins/pytorch/common_utils.sh
2022-09-27T15:28:21.0091071Z +++ declare -f -t trap_add
2022-09-27T15:28:21.0095527Z ++ set -ex
2022-09-27T15:28:21.0095883Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-09-27T15:28:21.0096097Z ++ BUILD_TEST_LIBTORCH=0
2022-09-27T15:28:21.0096559Z ++ [[ distributed == *xla* ]]
2022-09-27T15:28:21.0096933Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]]
2022-09-27T15:28:21.0097255Z ++ [[ linux-focal-py3.7-gcc7 == *linux-bionic* ]]
2022-09-27T15:28:21.0097564Z ++ [[ linux-focal-py3.7-gcc7 == *linux-focal* ]]
2022-09-27T15:28:21.0097778Z ++ which conda
2022-09-27T15:28:21.0104029Z /opt/conda/bin/conda
2022-09-27T15:28:21.0104710Z ++ conda install -q -y cmake
2022-09-27T15:28:23.0574547Z Collecting package metadata (current_repodata.json): ...working... done
2022-09-27T15:28:23.5212467Z Solving environment: ...working... done
2022-09-27T15:28:23.6110577Z 
2022-09-27T15:28:23.6110778Z ## Package Plan ##
2022-09-27T15:28:23.6111043Z 
2022-09-27T15:28:23.6111222Z   environment location: /opt/conda
2022-09-27T15:28:23.6111480Z 
2022-09-27T15:28:23.6111660Z   added / updated specs:
2022-09-27T15:28:23.6112213Z     - cmake
2022-09-27T15:28:23.6112328Z 
2022-09-27T15:28:23.6112334Z 
2022-09-27T15:28:23.6112465Z The following packages will be downloaded:
2022-09-27T15:28:23.6112624Z 
2022-09-27T15:28:23.6112735Z     package                    |            build
2022-09-27T15:28:23.6113049Z     ---------------------------|-----------------
2022-09-27T15:28:23.6113381Z     bzip2-1.0.8                |       h7b6447c_0          78 KB
2022-09-27T15:28:23.6113707Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-09-27T15:28:23.6114059Z     certifi-2022.9.14          |   py37h06a4308_0         155 KB
2022-09-27T15:28:23.6114389Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-09-27T15:28:23.6114794Z     conda-22.9.0               |   py37h06a4308_0         878 KB
2022-09-27T15:28:23.6115116Z     expat-2.4.4                |       h295c915_0         169 KB
2022-09-27T15:28:23.6115438Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-09-27T15:28:23.6115761Z     libcurl-7.82.0             |       h0b77cf5_0         342 KB
2022-09-27T15:28:23.6116075Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-09-27T15:28:23.6116397Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-09-27T15:28:23.6116728Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-09-27T15:28:23.6117048Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-09-27T15:28:23.6117365Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-09-27T15:28:23.6117859Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-09-27T15:28:23.6118166Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-09-27T15:28:23.6118555Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-09-27T15:28:23.6118894Z     ------------------------------------------------------------
2022-09-27T15:28:23.6119177Z                                            Total:        13.0 MB
2022-09-27T15:28:23.6119307Z 
2022-09-27T15:28:23.6119436Z The following NEW packages will be INSTALLED:
2022-09-27T15:28:23.6119596Z 
2022-09-27T15:28:23.6131048Z   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
2022-09-27T15:28:23.6131502Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-09-27T15:28:23.6131903Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-09-27T15:28:23.6132279Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-09-27T15:28:23.6132671Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-09-27T15:28:23.6133060Z   libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
2022-09-27T15:28:23.6133457Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-09-27T15:28:23.6133858Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-09-27T15:28:23.6134263Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-09-27T15:28:23.6134673Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-09-27T15:28:23.6135051Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-09-27T15:28:23.6135432Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-09-27T15:28:23.6135812Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-09-27T15:28:23.6136177Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-09-27T15:28:23.6136343Z 
2022-09-27T15:28:23.6136462Z The following packages will be UPDATED:
2022-09-27T15:28:23.6136614Z 
2022-09-27T15:28:23.6136840Z   certifi                          2022.6.15-py37h06a4308_0 --> 2022.9.14-py37h06a4308_0
2022-09-27T15:28:23.6137223Z   conda                               4.14.0-py37h06a4308_0 --> 22.9.0-py37h06a4308_0
2022-09-27T15:28:23.6137398Z 
2022-09-27T15:28:23.6137402Z 
2022-09-27T15:28:24.6735116Z Preparing transaction: ...working... done
2022-09-27T15:28:25.0702695Z Verifying transaction: ...working... done
2022-09-27T15:28:26.0691716Z Executing transaction: ...working... done
2022-09-27T15:28:26.1754709Z Retrieving notices: ...working... done
2022-09-27T15:28:26.3441076Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]]
2022-09-27T15:28:26.3441466Z + echo 'Environment variables'
2022-09-27T15:28:26.3441693Z Environment variables
2022-09-27T15:28:26.3441965Z + env
2022-09-27T15:28:26.3448407Z INSTALLED_DB=yes
2022-09-27T15:28:26.3449129Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:28:26.3449811Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2022-09-27T15:28:26.3450326Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-09-27T15:28:26.3450738Z HOSTNAME=89d4c2f4be45
2022-09-27T15:28:26.3452763Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_41b2f7f6-cc09-4037-81ae-8addc13b0a08
2022-09-27T15:28:26.3453374Z GITHUB_ACTION=__self
2022-09-27T15:28:26.3453775Z GITHUB_RUN_NUMBER=50832
2022-09-27T15:28:26.3454116Z TEST_CONFIG=distributed
2022-09-27T15:28:26.3454627Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-09-27T15:28:26.3455091Z GITHUB_TRIGGERING_ACTOR=albanD
2022-09-27T15:28:26.3455492Z GITHUB_REF_TYPE=branch
2022-09-27T15:28:26.3455901Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-09-27T15:28:26.3456322Z BASE_SHA=76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:28:26.3456547Z XLA_CUDA=
2022-09-27T15:28:26.3457113Z ***
2022-09-27T15:28:26.3457327Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-09-27T15:28:26.3457559Z GITHUB_ACTIONS=true
2022-09-27T15:28:26.3457786Z SHA1=52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:28:26.3458289Z GITHUB_SHA=1faa2af6dbb8dd899ab20874e9966185467c5883
2022-09-27T15:28:26.3458538Z UCC_HOME=/usr
2022-09-27T15:28:26.3463285Z GITHUB_REF=refs/pull/85462/merge
2022-09-27T15:28:26.3463663Z SHARD_NUMBER=2
2022-09-27T15:28:26.3464235Z GITHUB_REF_PROTECTED=false
2022-09-27T15:28:26.3464434Z HOME=/var/lib/jenkins
2022-09-27T15:28:26.3464658Z GITHUB_API_URL=https://api.github.com
2022-09-27T15:28:26.3464879Z INSTALLED_THRIFT=
2022-09-27T15:28:26.3465038Z UCX_COMMIT=
2022-09-27T15:28:26.3465224Z SCCACHE_S3_KEY_PREFIX=pull
2022-09-27T15:28:26.3465412Z NUM_TEST_SHARDS=2
2022-09-27T15:28:26.3510234Z UCX_HOME=/usr
2022-09-27T15:28:26.3510724Z PYTORCH_RETRY_TEST_CASES=1
2022-09-27T15:28:26.3511482Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_41b2f7f6-cc09-4037-81ae-8addc13b0a08
2022-09-27T15:28:26.3512265Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-09-27T15:28:26.3512707Z GITHUB_EVENT_NAME=pull_request
2022-09-27T15:28:26.3513019Z GITHUB_RUN_ID=3133193930
2022-09-27T15:28:26.3513745Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_41b2f7f6-cc09-4037-81ae-8addc13b0a08
2022-09-27T15:28:26.3514241Z GITHUB_ACTOR=kongzii
2022-09-27T15:28:26.3514533Z PR_NUMBER=85462
2022-09-27T15:28:26.3514824Z DESIRED_CUDA=
2022-09-27T15:28:26.3515105Z GITHUB_RUN_ATTEMPT=2
2022-09-27T15:28:26.3515398Z VALGRIND=ON
2022-09-27T15:28:26.3515770Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-09-27T15:28:26.3516127Z TERM=xterm
2022-09-27T15:28:26.3516407Z INSTALLED_VISION=yes
2022-09-27T15:28:26.3516690Z BRANCH=pull/85462
2022-09-27T15:28:26.3516991Z OPENSSL_ROOT_DIR=/opt/openssl
2022-09-27T15:28:26.3517299Z CUDA_PATH=/usr/local/cuda
2022-09-27T15:28:26.3517951Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-09-27T15:28:26.3518461Z GITHUB_SERVER_URL=https://github.com
2022-09-27T15:28:26.3518779Z UCC_COMMIT=
2022-09-27T15:28:26.3524614Z INSTALLED_ANDROID=
2022-09-27T15:28:26.3525062Z SHLVL=1
2022-09-27T15:28:26.3525336Z MAX_JOBS=6
2022-09-27T15:28:26.3526226Z COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting
2022-09-27T15:28:26.3527229Z GITHUB_REF_NAME=85462/merge
2022-09-27T15:28:26.3527861Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:28:26.3528306Z GITHUB_JOB=test
2022-09-27T15:28:26.3528636Z GITHUB_REPOSITORY=pytorch/pytorch
2022-09-27T15:28:26.3528955Z GITHUB_RETENTION_DAYS=90
2022-09-27T15:28:26.3529277Z OPENSSL_DIR=/opt/openssl
2022-09-27T15:28:26.3529541Z GITHUB_ACTION_REPOSITORY=
2022-09-27T15:28:26.3530058Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:28:26.3530601Z GITHUB_BASE_REF=master
2022-09-27T15:28:26.3530843Z CI=true
2022-09-27T15:28:26.3531122Z GITHUB_REPOSITORY_OWNER=pytorch
2022-09-27T15:28:26.3531457Z INSTALLED_PROTOBUF=yes
2022-09-27T15:28:26.3531782Z GITHUB_HEAD_REF=cycliclr-memory-fix
2022-09-27T15:28:26.3531989Z GITHUB_ACTION_REF=
2022-09-27T15:28:26.3532259Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-09-27T15:28:26.3532492Z GITHUB_WORKFLOW=pull
2022-09-27T15:28:26.3532695Z DEBIAN_FRONTEND=noninteractive
2022-09-27T15:28:26.3535567Z PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```
2022-09-27T15:28:26.3537383Z _=/usr/bin/env
2022-09-27T15:28:26.3537596Z + echo 'Testing pytorch'
2022-09-27T15:28:26.3537769Z Testing pytorch
2022-09-27T15:28:26.3537965Z + export LANG=C.UTF-8
2022-09-27T15:28:26.3538155Z + LANG=C.UTF-8
2022-09-27T15:28:26.3538313Z + PR_NUMBER=85462
2022-09-27T15:28:26.3538506Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-09-27T15:28:26.3538725Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-09-27T15:28:26.3538984Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-09-27T15:28:26.3540796Z + [[ distributed == \s\l\o\w ]]
2022-09-27T15:28:26.3541951Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]]
2022-09-27T15:28:26.3542256Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-09-27T15:28:26.3542527Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-09-27T15:28:26.3542789Z + [[ linux-focal-py3.7-gcc7 == *cuda11* ]]
2022-09-27T15:28:26.3543015Z + [[ distributed == *crossref* ]]
2022-09-27T15:28:26.3543216Z + [[ distributed == *dynamo* ]]
2022-09-27T15:28:26.3543415Z + [[ -n 85462 ]]
2022-09-27T15:28:26.3543605Z + [[ -z '' ]]
2022-09-27T15:28:26.3543813Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:28:26.3544210Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:28:26.3544480Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-09-27T15:28:26.3544757Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]]
2022-09-27T15:28:26.3544992Z + pip_install --user ninja
2022-09-27T15:28:26.3545259Z + pip install --progress-bar off --user ninja
2022-09-27T15:28:26.7469831Z Collecting ninja
2022-09-27T15:28:26.7614523Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-09-27T15:28:27.4723100Z Installing collected packages: ninja
2022-09-27T15:28:27.4807979Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-09-27T15:28:27.4808510Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T15:28:27.4862142Z Successfully installed ninja-1.10.2.3
2022-09-27T15:28:27.5479687Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:28:27.5480366Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:28:27.5481346Z + [[ linux-focal-py3.7-gcc7 == *asan* ]]
2022-09-27T15:28:27.5481658Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-09-27T15:28:27.5481902Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-09-27T15:28:27.5484824Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]]
2022-09-27T15:28:27.5496359Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-09-27T15:28:27.5496909Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]]
2022-09-27T15:28:27.5499407Z + cd test
2022-09-27T15:28:27.5499781Z + python -c 'import torch; print(torch.__config__.show())'
2022-09-27T15:28:28.5653467Z PyTorch built with:
2022-09-27T15:28:28.5654229Z   - GCC 7.5
2022-09-27T15:28:28.5654448Z   - C++ Version: 201402
2022-09-27T15:28:28.5654864Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:28:28.5655299Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:28:28.5655610Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:28:28.5655877Z   - LAPACK is enabled (usually provided by MKL)
2022-09-27T15:28:28.5656120Z   - NNPACK is enabled
2022-09-27T15:28:28.5656352Z   - CPU capability usage: AVX2
2022-09-27T15:28:28.5658680Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-09-27T15:28:28.5660365Z 
2022-09-27T15:28:28.7228935Z + cd test
2022-09-27T15:28:28.7229323Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-09-27T15:28:29.7147121Z ATen/Parallel:
2022-09-27T15:28:29.7147548Z 	at::get_num_threads() : 4
2022-09-27T15:28:29.7147935Z 	at::get_num_interop_threads() : 4
2022-09-27T15:28:29.7148199Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:28:29.7148418Z 	omp_get_max_threads() : 4
2022-09-27T15:28:29.7148930Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:28:29.7149227Z 	mkl_get_max_threads() : 4
2022-09-27T15:28:29.7149550Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:28:29.7149825Z std::thread::hardware_concurrency() : 8
2022-09-27T15:28:29.7150036Z Environment variables:
2022-09-27T15:28:29.7150222Z 	OMP_NUM_THREADS : [not set]
2022-09-27T15:28:29.7150420Z 	MKL_NUM_THREADS : [not set]
2022-09-27T15:28:29.7150637Z ATen parallel backend: OpenMP
2022-09-27T15:28:29.7150751Z 
2022-09-27T15:28:29.8740477Z + [[ distributed == *deploy* ]]
2022-09-27T15:28:29.8740913Z + [[ distributed == *backward* ]]
2022-09-27T15:28:29.8741313Z + [[ distributed == *xla* ]]
2022-09-27T15:28:29.8741614Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-09-27T15:28:29.8741993Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-09-27T15:28:29.8742268Z + [[ distributed == distributed ]]
2022-09-27T15:28:29.8742532Z + install_torchdynamo
2022-09-27T15:28:29.8742740Z + local commit
2022-09-27T15:28:29.8743219Z ++ get_pinned_commit torchdynamo
2022-09-27T15:28:29.8743494Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-09-27T15:28:29.8755283Z + commit=41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:29.8756115Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:29.8756873Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:30.2192793Z Collecting git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:30.2197961Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 41c44bc1d080d6cf063419a4166732b983b84eef) to /tmp/pip-req-build-6m2jb80e
2022-09-27T15:28:30.2227029Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-6m2jb80e
2022-09-27T15:28:30.8653668Z   Running command git rev-parse -q --verify 'sha^41c44bc1d080d6cf063419a4166732b983b84eef'
2022-09-27T15:28:30.8686676Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:31.1217130Z   Running command git checkout -q 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:31.3841427Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:28:33.0696778Z   Preparing metadata (setup.py) ... [?25l- done
2022-09-27T15:28:33.0768485Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+git52424e2)
2022-09-27T15:28:33.0772072Z Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.18.5)
2022-09-27T15:28:33.1118842Z Collecting tabulate
2022-09-27T15:28:33.1264358Z   Downloading tabulate-0.8.10-py3-none-any.whl (29 kB)
2022-09-27T15:28:33.1310164Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (6.0)
2022-09-27T15:28:33.1313719Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1)
2022-09-27T15:28:33.1335326Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-09-27T15:28:33.1358505Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-09-27T15:28:33.1452889Z Building wheels for collected packages: torchdynamo
2022-09-27T15:28:36.6177584Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-09-27T15:28:36.6255834Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp37-cp37m-linux_x86_64.whl size=2593898 sha256=e1fbaa39c4a8b2f1c53856bd7ae1e22434831ed7357da1b35e216657ecab8dea
2022-09-27T15:28:36.6257242Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/77/0f/de/99455111d0619b9c9ed3eeb8e83431c83a6c69394197cf5fb8
2022-09-27T15:28:36.6285125Z Successfully built torchdynamo
2022-09-27T15:28:37.3224482Z Installing collected packages: tabulate, torchdynamo
2022-09-27T15:28:37.5739633Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0
2022-09-27T15:28:37.6510157Z + test_distributed
2022-09-27T15:28:37.6510728Z + echo 'Testing distributed python tests'
2022-09-27T15:28:37.6511186Z Testing distributed python tests
2022-09-27T15:28:37.6511543Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose
2022-09-27T15:28:39.0344148Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-09-27T15:28:39.0923594Z Ignoring disabled issues:  []
2022-09-27T15:28:39.1039627Z Found test time stats from artifacts
2022-09-27T15:28:39.1041093Z Selected tests:
2022-09-27T15:28:39.1041418Z  distributed/test_distributed_spawn
2022-09-27T15:28:39.1041792Z  distributed/rpc/test_faulty_agent
2022-09-27T15:28:39.1042135Z  distributed/test_c10d_gloo
2022-09-27T15:28:39.1042472Z  distributed/fsdp/test_fsdp_summon_full_params
2022-09-27T15:28:39.1042815Z  distributed/fsdp/test_fsdp_core
2022-09-27T15:28:39.1043190Z  distributed/fsdp/test_fsdp_mixed_precision
2022-09-27T15:28:39.1043596Z  distributed/test_c10d_pypg
2022-09-27T15:28:39.1043978Z  distributed/optim/test_zero_redundancy_optimizer
2022-09-27T15:28:39.1044433Z  distributed/fsdp/test_fsdp_comm_hooks
2022-09-27T15:28:39.1044913Z  distributed/fsdp/test_fsdp_grad_acc
2022-09-27T15:28:39.1045147Z  distributed/test_pg_wrapper
2022-09-27T15:28:39.1045355Z  distributed/test_c10d_spawn_gloo
2022-09-27T15:28:39.1045647Z  distributed/fsdp/test_fsdp_comm
2022-09-27T15:28:39.1045869Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-09-27T15:28:39.1046105Z  distributed/fsdp/test_fsdp_checkpoint
2022-09-27T15:28:39.1046320Z  distributed/fsdp/test_fsdp_meta
2022-09-27T15:28:39.1046545Z  distributed/fsdp/test_fsdp_ignored_modules
2022-09-27T15:28:39.1046820Z  distributed/algorithms/quantization/test_quantization
2022-09-27T15:28:39.1047073Z  distributed/test_c10d_object_collectives
2022-09-27T15:28:39.1047385Z  distributed/fsdp/test_fsdp_apply
2022-09-27T15:28:39.1047804Z  distributed/_shard/checkpoint/test_checkpoint
2022-09-27T15:28:39.1048160Z  distributed/fsdp/test_fsdp_memory
2022-09-27T15:28:39.1048375Z  distributed/fsdp/test_fsdp_input
2022-09-27T15:28:39.1048577Z  distributed/rpc/test_share_memory
2022-09-27T15:28:39.1048814Z  distributed/fsdp/test_fsdp_multiple_forward
2022-09-27T15:28:39.1049057Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-09-27T15:28:39.1049287Z  distributed/fsdp/test_fsdp_traversal
2022-09-27T15:28:39.1049511Z  distributed/fsdp/test_flatten_params_wrapper
2022-09-27T15:28:39.1049741Z  distributed/elastic/utils/util_test
2022-09-27T15:28:39.1049969Z  distributed/fsdp/test_checkpoint_wrapper
2022-09-27T15:28:39.1050214Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-09-27T15:28:39.1050461Z  distributed/elastic/utils/logging_test
2022-09-27T15:28:39.1050679Z  distributed/fsdp/test_utils
2022-09-27T15:28:39.1050877Z  distributed/nn/jit/test_instantiator
2022-09-27T15:28:39.1051096Z  distributed/elastic/metrics/api_test
2022-09-27T15:28:39.1051309Z  distributed/test_data_parallel
2022-09-27T15:28:39.1051539Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-09-27T15:28:39.1051809Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-09-27T15:28:39.1052067Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-09-27T15:28:39.1052310Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-09-27T15:28:39.1052549Z  distributed/_shard/test_partial_tensor
2022-09-27T15:28:39.1052800Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-09-27T15:28:39.1053062Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-09-27T15:28:39.1053303Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-09-27T15:28:39.1053568Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-09-27T15:28:39.1053837Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-09-27T15:28:39.1054070Z  distributed/_shard/test_replicated_tensor
2022-09-27T15:28:39.1054297Z  distributed/elastic/events/lib_test
2022-09-27T15:28:39.1054519Z  distributed/fsdp/test_shard_utils
2022-09-27T15:28:39.1054740Z  distributed/pipeline/sync/skip/test_gpipe
2022-09-27T15:28:39.1054982Z  distributed/pipeline/sync/skip/test_leak
2022-09-27T15:28:39.1055228Z  distributed/pipeline/sync/skip/test_stash_pop
2022-09-27T15:28:39.1055476Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-09-27T15:28:39.1055723Z  distributed/pipeline/sync/test_bugs
2022-09-27T15:28:39.1055951Z  distributed/pipeline/sync/test_copy
2022-09-27T15:28:39.1056177Z  distributed/pipeline/sync/test_dependency
2022-09-27T15:28:39.1056423Z  distributed/pipeline/sync/test_microbatch
2022-09-27T15:28:39.1056657Z  distributed/pipeline/sync/test_pipe
2022-09-27T15:28:39.1056885Z  distributed/pipeline/sync/test_stream
2022-09-27T15:28:39.1057100Z  distributed/pipeline/sync/test_worker
2022-09-27T15:28:39.1186225Z Prioritized test from test file changes.
2022-09-27T15:28:39.1186710Z reordering tests for PR:
2022-09-27T15:28:39.1187318Z prioritized: ['distributed/test_c10d_gloo', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_checkpoint_wrapper']
2022-09-27T15:28:39.1191639Z the rest: ['distributed/test_distributed_spawn', 'distributed/rpc/test_faulty_agent', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_core', 'distributed/test_c10d_pypg', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/test_pg_wrapper', 'distributed/test_c10d_spawn_gloo', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/algorithms/quantization/test_quantization', 'distributed/test_c10d_object_collectives', 'distributed/fsdp/test_fsdp_apply', 'distributed/_shard/checkpoint/test_checkpoint', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_input', 'distributed/rpc/test_share_memory', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/elastic/utils/util_test', 'distributed/_shard/checkpoint/test_file_system_checkpoint', 'distributed/elastic/utils/logging_test', 'distributed/fsdp/test_utils', 'distributed/nn/jit/test_instantiator', 'distributed/elastic/metrics/api_test', 'distributed/test_data_parallel', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/test_partial_tensor', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/test_replicated_tensor', 'distributed/elastic/events/lib_test', 'distributed/fsdp/test_shard_utils', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_worker']
2022-09-27T15:28:39.1194296Z 
2022-09-27T15:28:39.1194717Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-09-27T15:28:39.1890784Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-09-27T15:28:39.2051126Z Running distributed/test_c10d_gloo ... [2022-09-27 15:28:39.204883]
2022-09-27T15:28:39.2051959Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:28:39.204923]
2022-09-27T15:28:40.3842703Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-09-27T15:28:40.3844291Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-09-27T15:28:40.3844693Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-09-27T15:28:40.3845303Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-09-27T15:28:40.3845789Z test_gloo_rank_membership (__main__.CommTest)
2022-09-27T15:28:40.3846165Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-09-27T15:28:40.3846680Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-09-27T15:28:40.3847132Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-09-27T15:28:40.3847520Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-09-27T15:28:40.3847932Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-09-27T15:28:40.3848326Z test_tensor_dtype_complex (__main__.CommTest)
2022-09-27T15:28:40.3848644Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-09-27T15:28:40.3849844Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-09-27T15:28:40.3850864Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:28:40.3851126Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:28:40.3851389Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:28:40.3851630Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:28:40.3851884Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:28:40.3852138Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:28:40.3852390Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:28:40.3852664Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:28:40.3852933Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-09-27T15:28:40.3853182Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:28:40.3853430Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:28:40.3857771Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-09-27T15:28:40.3862057Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3862422Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3862791Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3863139Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3863521Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3864166Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3864585Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3864936Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3865293Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3865740Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3866126Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3866495Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3866878Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3867236Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3867565Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3867903Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3868234Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3868558Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3868873Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3869307Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3869668Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3870054Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3870421Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3870788Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3871157Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3871487Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3871803Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3872117Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3872435Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3872751Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3873073Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3873405Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3873704Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3874010Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3874319Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3874629Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3874955Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T15:28:40.3875232Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:28:40.3879320Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-09-27T15:28:40.3883321Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3883608Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3899203Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3899655Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3900014Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3900457Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3900853Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3901247Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3901626Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3902027Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3902436Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3902877Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3903284Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3903691Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3904338Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3904713Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3905069Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3905374Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3905726Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3906014Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3906297Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3906566Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3906845Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3907122Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3907375Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3907656Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3907928Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3908178Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3908448Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3908717Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3909163Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3909434Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3909703Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3910044Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3910310Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3910579Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3910846Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3911093Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3911361Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3911629Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3911904Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3912168Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3912440Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3912712Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3912960Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3913234Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3913513Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3913784Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3914075Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3914372Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:28:40.3915039Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-09-27T15:28:40.3915652Z test_forward_backward (__main__.ReducerTest)
2022-09-27T15:28:40.3915893Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-09-27T15:28:40.3916173Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-09-27T15:28:40.3916443Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-09-27T15:28:40.3916685Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-09-27T15:28:40.3916944Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-09-27T15:28:40.3917260Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-09-27T15:28:40.3917552Z test_logging_init (__main__.RendezvousEnvTest)
2022-09-27T15:28:40.3917797Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:28:40.3918112Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-09-27T15:28:40.3918428Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-09-27T15:28:41.4344892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:41.4345550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:41.4346418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:41.4346982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:41.5696756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:41.5707110Z 
2022-09-27T15:28:41.5707631Z Running tests...
2022-09-27T15:28:41.5708045Z ----------------------------------------------------------------------
2022-09-27T15:28:41.8033210Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 703
2022-09-27T15:28:41.8057472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 704
2022-09-27T15:28:42.8901955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:42.8902759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:42.8904007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:42.8904562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:42.8919945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:42.8920489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:42.8923352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:42.8923778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:43.0220764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:43.0247482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:43.3094719Z ok (1.738s)
2022-09-27T15:28:43.3095156Z 
2022-09-27T15:28:43.3095664Z ----------------------------------------------------------------------
2022-09-27T15:28:43.3096030Z Ran 1 test in 1.739s
2022-09-27T15:28:43.3096152Z 
2022-09-27T15:28:43.3096239Z OK
2022-09-27T15:28:43.3096348Z 
2022-09-27T15:28:43.3096457Z Generating XML reports...
2022-09-27T15:28:43.3127742Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152841.xml
2022-09-27T15:28:44.6327399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:44.6327930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:44.6329203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:44.6329750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:44.7670079Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:44.7679350Z 
2022-09-27T15:28:44.7679465Z Running tests...
2022-09-27T15:28:44.7679887Z ----------------------------------------------------------------------
2022-09-27T15:28:45.0004142Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 742
2022-09-27T15:28:45.0029127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 743
2022-09-27T15:28:46.0677569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:46.0677988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:46.0679776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:46.0680334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:46.1216524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:46.1217139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:46.1218089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:46.1218680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:46.1997838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:46.2537523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:46.5065873Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:28:46.5066283Z 
2022-09-27T15:28:46.5066595Z ----------------------------------------------------------------------
2022-09-27T15:28:46.5066847Z Ran 1 test in 1.739s
2022-09-27T15:28:46.5067047Z 
2022-09-27T15:28:46.5067120Z OK (skipped=1)
2022-09-27T15:28:46.5067228Z 
2022-09-27T15:28:46.5067314Z Generating XML reports...
2022-09-27T15:28:46.5098925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152844.xml
2022-09-27T15:28:47.8321205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:47.8321580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:47.8323747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:47.8324130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:47.9666674Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:47.9676662Z 
2022-09-27T15:28:47.9676862Z Running tests...
2022-09-27T15:28:47.9677207Z ----------------------------------------------------------------------
2022-09-27T15:28:48.1997689Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 775
2022-09-27T15:28:48.2021967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 776
2022-09-27T15:28:49.2894885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:49.2895443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:49.2897531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:49.2898101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:49.3039068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:49.3039692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:49.3041869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:49.3042461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:49.4225155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:49.4348987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:49.4533566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:28:49.4534184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:28:49.4535051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:28:49.4535754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:28:49.7061250Z ok (1.738s)
2022-09-27T15:28:49.7061494Z 
2022-09-27T15:28:49.7061926Z ----------------------------------------------------------------------
2022-09-27T15:28:49.7062187Z Ran 1 test in 1.738s
2022-09-27T15:28:49.7062296Z 
2022-09-27T15:28:49.7062358Z OK
2022-09-27T15:28:49.7062450Z 
2022-09-27T15:28:49.7062584Z Generating XML reports...
2022-09-27T15:28:49.7095226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152847.xml
2022-09-27T15:28:51.0638159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:51.0638811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:51.0640279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:51.0640855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:51.2000865Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:51.2010590Z 
2022-09-27T15:28:51.2011016Z Running tests...
2022-09-27T15:28:51.2011413Z ----------------------------------------------------------------------
2022-09-27T15:28:51.4352295Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 814
2022-09-27T15:28:51.4376999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 815
2022-09-27T15:28:52.5248965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:52.5249494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:52.5250711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:52.5251144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:52.5440356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:52.5440711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:52.5443606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:52.5443974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:52.6574659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:52.6755844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:52.9416311Z skip: Need at least 2 CUDA devices (1.740s)
2022-09-27T15:28:52.9416603Z 
2022-09-27T15:28:52.9417035Z ----------------------------------------------------------------------
2022-09-27T15:28:52.9417286Z Ran 1 test in 1.740s
2022-09-27T15:28:52.9417410Z 
2022-09-27T15:28:52.9417482Z OK (skipped=1)
2022-09-27T15:28:52.9417590Z 
2022-09-27T15:28:52.9417675Z Generating XML reports...
2022-09-27T15:28:52.9448952Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152851.xml
2022-09-27T15:28:54.3331552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:54.3331934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:54.3333790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:54.3334150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:54.4675210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:54.4684850Z 
2022-09-27T15:28:54.4685000Z Running tests...
2022-09-27T15:28:54.4685440Z ----------------------------------------------------------------------
2022-09-27T15:28:54.6993430Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 847
2022-09-27T15:28:54.7018096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 848
2022-09-27T15:28:55.7760293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:55.7760698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:55.7762637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:55.7763109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:55.7788454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:55.7788853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:55.7791886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:55.7792356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:55.9137462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:55.9243851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:56.2057691Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:28:56.2057895Z 
2022-09-27T15:28:56.2058271Z ----------------------------------------------------------------------
2022-09-27T15:28:56.2058567Z Ran 1 test in 1.737s
2022-09-27T15:28:56.2058699Z 
2022-09-27T15:28:56.2058816Z OK (skipped=1)
2022-09-27T15:28:56.2058976Z 
2022-09-27T15:28:56.2059078Z Generating XML reports...
2022-09-27T15:28:56.2090913Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152854.xml
2022-09-27T15:28:57.6499217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:57.6499639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:57.6501428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:57.6502045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:57.7874776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:28:57.7884101Z 
2022-09-27T15:28:57.7884270Z Running tests...
2022-09-27T15:28:57.7884758Z ----------------------------------------------------------------------
2022-09-27T15:28:58.0238959Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 880
2022-09-27T15:28:58.0263252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 881
2022-09-27T15:28:59.1300868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:59.1302678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:59.1303469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:59.1304014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:59.1470951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:28:59.1471449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:28:59.1473834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:28:59.1474370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:28:59.2631386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:28:59.2819920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:28:59.5301832Z skip: Need at least 2 CUDA devices (1.741s)
2022-09-27T15:28:59.5302140Z 
2022-09-27T15:28:59.5302735Z ----------------------------------------------------------------------
2022-09-27T15:28:59.5303118Z Ran 1 test in 1.742s
2022-09-27T15:28:59.5303253Z 
2022-09-27T15:28:59.5303346Z OK (skipped=1)
2022-09-27T15:28:59.5303471Z 
2022-09-27T15:28:59.5303575Z Generating XML reports...
2022-09-27T15:28:59.5334881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152857.xml
2022-09-27T15:29:00.8538410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:00.8538889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:00.8539527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:00.8539869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:00.9888501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:00.9898386Z 
2022-09-27T15:29:00.9898920Z Running tests...
2022-09-27T15:29:00.9899343Z ----------------------------------------------------------------------
2022-09-27T15:29:01.2351208Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 913
2022-09-27T15:29:01.2376179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 914
2022-09-27T15:29:02.3108262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:02.3108957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:02.3110185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:02.3110817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:02.3171223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:02.3171617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:02.3174423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:02.3174804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:02.4467223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:02.4488358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:02.7417050Z skip: Need at least 4 CUDA devices (1.752s)
2022-09-27T15:29:02.7417352Z 
2022-09-27T15:29:02.7417848Z ----------------------------------------------------------------------
2022-09-27T15:29:02.7418346Z Ran 1 test in 1.752s
2022-09-27T15:29:02.7418583Z 
2022-09-27T15:29:02.7418746Z OK (skipped=1)
2022-09-27T15:29:02.7418921Z 
2022-09-27T15:29:02.7419013Z Generating XML reports...
2022-09-27T15:29:02.7452901Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152900.xml
2022-09-27T15:29:04.1122027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:04.1122649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:04.1125096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:04.1125664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:04.2478226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:04.2487681Z 
2022-09-27T15:29:04.2488291Z Running tests...
2022-09-27T15:29:04.2488876Z ----------------------------------------------------------------------
2022-09-27T15:29:04.4813260Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 946
2022-09-27T15:29:04.4838092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 947
2022-09-27T15:29:05.5413702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:05.5414394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:05.5415851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:05.5416456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:05.5507838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:05.5508539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:05.5510697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:05.5511344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:05.6734943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:05.6822203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:05.9875295Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:29:05.9876247Z 
2022-09-27T15:29:05.9876817Z ----------------------------------------------------------------------
2022-09-27T15:29:05.9877070Z Ran 1 test in 1.739s
2022-09-27T15:29:05.9877185Z 
2022-09-27T15:29:05.9877263Z OK (skipped=1)
2022-09-27T15:29:05.9877369Z 
2022-09-27T15:29:05.9877457Z Generating XML reports...
2022-09-27T15:29:05.9909236Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152904.xml
2022-09-27T15:29:07.3417938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:07.3418501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:07.3420587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:07.3421163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:07.4765822Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:07.4775647Z 
2022-09-27T15:29:07.4776079Z Running tests...
2022-09-27T15:29:07.4776491Z ----------------------------------------------------------------------
2022-09-27T15:29:07.7096927Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 979
2022-09-27T15:29:07.7121587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 980
2022-09-27T15:29:08.7955560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:08.7955944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:08.7957633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:08.7957982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:08.8141715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:08.8142092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:08.8144854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:08.8145206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:08.9272671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:08.9459733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:09.2158805Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:29:09.2159228Z 
2022-09-27T15:29:09.2159736Z ----------------------------------------------------------------------
2022-09-27T15:29:09.2160078Z Ran 1 test in 1.738s
2022-09-27T15:29:09.2160191Z 
2022-09-27T15:29:09.2160465Z OK (skipped=1)
2022-09-27T15:29:09.2160576Z 
2022-09-27T15:29:09.2160661Z Generating XML reports...
2022-09-27T15:29:09.2191723Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152907.xml
2022-09-27T15:29:10.5558127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:10.5558515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:10.5560157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:10.5560501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:10.6904747Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:10.6914841Z 
2022-09-27T15:29:10.6914954Z Running tests...
2022-09-27T15:29:10.6915390Z ----------------------------------------------------------------------
2022-09-27T15:29:10.9247791Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1012
2022-09-27T15:29:10.9272896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1013
2022-09-27T15:29:11.9850331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:11.9850812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:11.9852356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:11.9852890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:12.0090964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:12.0091318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:12.0093839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:12.0094198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:12.1182888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:12.1446855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:12.4311087Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:29:12.4311394Z 
2022-09-27T15:29:12.4311732Z ----------------------------------------------------------------------
2022-09-27T15:29:12.4312067Z Ran 1 test in 1.740s
2022-09-27T15:29:12.4312169Z 
2022-09-27T15:29:12.4312244Z OK (skipped=1)
2022-09-27T15:29:12.4312352Z 
2022-09-27T15:29:12.4312439Z Generating XML reports...
2022-09-27T15:29:12.4343766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152910.xml
2022-09-27T15:29:13.7972890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:13.7973316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:13.7974792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:13.7975238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:13.9327676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:13.9336727Z 
2022-09-27T15:29:13.9336899Z Running tests...
2022-09-27T15:29:13.9337391Z ----------------------------------------------------------------------
2022-09-27T15:29:14.1661613Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1045
2022-09-27T15:29:14.1686652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1046
2022-09-27T15:29:15.2168424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:15.2169312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:15.2170054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:15.2170611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:15.2418134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:15.2418789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:15.2420465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:15.2421144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:15.3491583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:15.3766963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:15.6728305Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:29:15.6728610Z 
2022-09-27T15:29:15.6728946Z ----------------------------------------------------------------------
2022-09-27T15:29:15.6729232Z Ran 1 test in 1.739s
2022-09-27T15:29:15.6729365Z 
2022-09-27T15:29:15.6729457Z OK (skipped=1)
2022-09-27T15:29:15.6729569Z 
2022-09-27T15:29:15.6729675Z Generating XML reports...
2022-09-27T15:29:15.6765079Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152913.xml
2022-09-27T15:29:17.0148680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:17.0149082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:17.0150820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:17.0151212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:17.1494899Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:17.1504453Z 
2022-09-27T15:29:17.1504814Z Running tests...
2022-09-27T15:29:17.1505560Z ----------------------------------------------------------------------
2022-09-27T15:29:17.3832597Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1078
2022-09-27T15:29:17.3857865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1079
2022-09-27T15:29:18.4625309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:18.4626054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:18.4627883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:18.4628536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:18.4716006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:18.4716732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:18.4720825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:18.4721522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:18.6006514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:18.6134754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:18.6320086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:18.6320800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:18.6321905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:18.6322864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:18.6466602Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:18.6467687Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:18.6468944Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:18.6469976Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:18.8894961Z ok (1.739s)
2022-09-27T15:29:18.8895199Z 
2022-09-27T15:29:18.8895536Z ----------------------------------------------------------------------
2022-09-27T15:29:18.8895829Z Ran 1 test in 1.739s
2022-09-27T15:29:18.8895965Z 
2022-09-27T15:29:18.8896029Z OK
2022-09-27T15:29:18.8896138Z 
2022-09-27T15:29:18.8896253Z Generating XML reports...
2022-09-27T15:29:18.8927152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152917.xml
2022-09-27T15:29:20.2125268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:20.2125872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:20.2127303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:20.2127921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:20.3475356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:20.3484128Z 
2022-09-27T15:29:20.3484238Z Running tests...
2022-09-27T15:29:20.3484818Z ----------------------------------------------------------------------
2022-09-27T15:29:20.5804350Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1117
2022-09-27T15:29:20.5828167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1118
2022-09-27T15:29:21.6316323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:21.6316795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:21.6318572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:21.6319097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:21.6861111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:21.6861464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:21.6863770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:21.6864293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:21.7643968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:21.8217599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:22.0867060Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:29:22.0867394Z 
2022-09-27T15:29:22.0867858Z ----------------------------------------------------------------------
2022-09-27T15:29:22.0868379Z Ran 1 test in 1.738s
2022-09-27T15:29:22.0868560Z 
2022-09-27T15:29:22.0868677Z OK (skipped=1)
2022-09-27T15:29:22.0868849Z 
2022-09-27T15:29:22.0870121Z Generating XML reports...
2022-09-27T15:29:22.0901094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152920.xml
2022-09-27T15:29:23.4372684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:23.4373068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:23.4374853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:23.4375226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:23.5724126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:23.5734119Z 
2022-09-27T15:29:23.5734373Z Running tests...
2022-09-27T15:29:23.5735006Z ----------------------------------------------------------------------
2022-09-27T15:29:23.8081115Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1150
2022-09-27T15:29:23.8106750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1151
2022-09-27T15:29:24.8851020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:24.8851428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:24.8853398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:24.8853950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:24.9238331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:24.9238808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:24.9241545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:24.9242008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:25.0183295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:25.0554419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:25.0766684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:25.0767407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:25.0768039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:25.0768581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:25.0892063Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:25.0892837Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:25.0894011Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:25.0894757Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:25.0895539Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:25.0896420Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:25.0897084Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:25.0897606Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:25.4146279Z ok (1.841s)
2022-09-27T15:29:25.4146512Z 
2022-09-27T15:29:25.4147044Z ----------------------------------------------------------------------
2022-09-27T15:29:25.4147482Z Ran 1 test in 1.841s
2022-09-27T15:29:25.4147630Z 
2022-09-27T15:29:25.4147684Z OK
2022-09-27T15:29:25.4147776Z 
2022-09-27T15:29:25.4147868Z Generating XML reports...
2022-09-27T15:29:25.4180126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152923.xml
2022-09-27T15:29:26.7402197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:26.7402616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:26.7403695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:26.7404050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:26.8744570Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:26.8753781Z 
2022-09-27T15:29:26.8754471Z Running tests...
2022-09-27T15:29:26.8755144Z ----------------------------------------------------------------------
2022-09-27T15:29:27.1081641Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1189
2022-09-27T15:29:27.1106754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1190
2022-09-27T15:29:28.1883688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:28.1884058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:28.1885927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:28.1886342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:28.1962641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:28.1962986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:28.1966308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:28.1966701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:28.3211358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:28.3319156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:28.6144466Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:29:28.6144738Z 
2022-09-27T15:29:28.6145259Z ----------------------------------------------------------------------
2022-09-27T15:29:28.6145639Z Ran 1 test in 1.739s
2022-09-27T15:29:28.6145754Z 
2022-09-27T15:29:28.6145814Z OK (skipped=1)
2022-09-27T15:29:28.6145925Z 
2022-09-27T15:29:28.6146011Z Generating XML reports...
2022-09-27T15:29:28.6177407Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152926.xml
2022-09-27T15:29:29.9999167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:29.9999930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:30.0001193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:30.0001642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:30.1344568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:30.1354161Z 
2022-09-27T15:29:30.1354377Z Running tests...
2022-09-27T15:29:30.1354757Z ----------------------------------------------------------------------
2022-09-27T15:29:30.3685248Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1222
2022-09-27T15:29:30.3711230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1223
2022-09-27T15:29:31.4573812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:31.4574491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:31.4575773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:31.4576152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:31.4660975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:31.4661420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:31.4663783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:31.4664342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:31.5891461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:31.5976881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:31.6189015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:31.6189453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:31.6190100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:31.6190648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:31.6313345Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:31.6313959Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:31.6314656Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:31.6315233Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:31.8749330Z ok (1.739s)
2022-09-27T15:29:31.8749614Z 
2022-09-27T15:29:31.8750095Z ----------------------------------------------------------------------
2022-09-27T15:29:31.8750522Z Ran 1 test in 1.739s
2022-09-27T15:29:31.8750731Z 
2022-09-27T15:29:31.8750848Z OK
2022-09-27T15:29:31.8751016Z 
2022-09-27T15:29:31.8751189Z Generating XML reports...
2022-09-27T15:29:31.8783679Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152930.xml
2022-09-27T15:29:33.2085167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:33.2085845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:33.2087594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:33.2087953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:33.3434350Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:33.3443551Z 
2022-09-27T15:29:33.3443703Z Running tests...
2022-09-27T15:29:33.3444133Z ----------------------------------------------------------------------
2022-09-27T15:29:33.5765400Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1261
2022-09-27T15:29:33.5789988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1262
2022-09-27T15:29:34.6219530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:34.6219992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:34.6221269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:34.6221681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:34.6478510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:34.6478859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:34.6481545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:34.6481903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:34.7549156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:34.7825308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:35.0827461Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:29:35.0827993Z 
2022-09-27T15:29:35.0828453Z ----------------------------------------------------------------------
2022-09-27T15:29:35.0828759Z Ran 1 test in 1.738s
2022-09-27T15:29:35.0828914Z 
2022-09-27T15:29:35.0828988Z OK (skipped=1)
2022-09-27T15:29:35.0829096Z 
2022-09-27T15:29:35.0829181Z Generating XML reports...
2022-09-27T15:29:35.0861127Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152933.xml
2022-09-27T15:29:36.4246166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:36.4246587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:36.4248102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:36.4248733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:36.5589745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:36.5598403Z 
2022-09-27T15:29:36.5598519Z Running tests...
2022-09-27T15:29:36.5599228Z ----------------------------------------------------------------------
2022-09-27T15:29:36.7919107Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1294
2022-09-27T15:29:36.7944657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1295
2022-09-27T15:29:37.8585725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:37.8586453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:37.8587443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:37.8588121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:37.8651214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:37.8651849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:37.8655073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:37.8655647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:37.9912664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:37.9974686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:38.0186532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:38.0187115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:38.0187775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:38.0188308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:38.0326086Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0327128Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0328207Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0329067Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0329733Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0330279Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0330929Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0331451Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0332096Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0332864Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0333550Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0334069Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0334956Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0335955Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.0337008Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:38.0337571Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:38.2981684Z ok (1.738s)
2022-09-27T15:29:38.2982013Z 
2022-09-27T15:29:38.2982714Z ----------------------------------------------------------------------
2022-09-27T15:29:38.2983211Z Ran 1 test in 1.738s
2022-09-27T15:29:38.2983424Z 
2022-09-27T15:29:38.2983571Z OK
2022-09-27T15:29:38.2983751Z 
2022-09-27T15:29:38.2984116Z Generating XML reports...
2022-09-27T15:29:38.3017475Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152936.xml
2022-09-27T15:29:39.6264222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:39.6264607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:39.6266997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:39.6267410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:39.7608431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:39.7618411Z 
2022-09-27T15:29:39.7618555Z Running tests...
2022-09-27T15:29:39.7619138Z ----------------------------------------------------------------------
2022-09-27T15:29:39.9935972Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1333
2022-09-27T15:29:39.9961282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1334
2022-09-27T15:29:41.0702285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:41.0702826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:41.0704735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:41.0705085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:41.0793511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:41.0794280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:41.0796005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:41.0796617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:41.2042189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:41.2163435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:41.5002998Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:29:41.5003311Z 
2022-09-27T15:29:41.5003918Z ----------------------------------------------------------------------
2022-09-27T15:29:41.5004289Z Ran 1 test in 1.738s
2022-09-27T15:29:41.5004422Z 
2022-09-27T15:29:41.5004515Z OK (skipped=1)
2022-09-27T15:29:41.5004641Z 
2022-09-27T15:29:41.5004737Z Generating XML reports...
2022-09-27T15:29:41.5036450Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152939.xml
2022-09-27T15:29:42.8551324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:42.8551974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:42.8553921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:42.8554304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:42.9896950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:42.9907384Z 
2022-09-27T15:29:42.9907654Z Running tests...
2022-09-27T15:29:42.9908254Z ----------------------------------------------------------------------
2022-09-27T15:29:43.2251051Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1366
2022-09-27T15:29:43.2276909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1367
2022-09-27T15:29:44.2955020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:44.2956414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:44.2957066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:44.2957556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:44.3192890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:44.3193487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:44.3195877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:44.3196447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:44.4287443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:44.4507415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:44.4700644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:44.4701298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:44.4702224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:44.4703067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:44.4927173Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:44.4927894Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:44.4928568Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:44.4929101Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:44.4929747Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:44.4930283Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:44.4931098Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:44.4931729Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:44.7314243Z ok (1.740s)
2022-09-27T15:29:44.7314517Z 
2022-09-27T15:29:44.7315074Z ----------------------------------------------------------------------
2022-09-27T15:29:44.7315356Z Ran 1 test in 1.741s
2022-09-27T15:29:44.7315476Z 
2022-09-27T15:29:44.7315525Z OK
2022-09-27T15:29:44.7315619Z 
2022-09-27T15:29:44.7315718Z Generating XML reports...
2022-09-27T15:29:44.7346845Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152942.xml
2022-09-27T15:29:46.0573814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:46.0574287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:46.0575063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:46.0575428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:46.1952323Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:46.1968189Z 
2022-09-27T15:29:46.1968333Z Running tests...
2022-09-27T15:29:46.1969042Z ----------------------------------------------------------------------
2022-09-27T15:29:46.4392826Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1405
2022-09-27T15:29:46.4419255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1406
2022-09-27T15:29:47.5007659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:47.5008169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:47.5008996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:47.5122823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:47.5123596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:47.5123928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:47.5126073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:47.5126579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:47.6336796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:47.6436118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:47.6549060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:29:47.6549721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:29:47.6550657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:47.6551555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:29:47.6684827Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:47.6685692Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:47.6686532Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:29:47.6687164Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:29:47.9455001Z ok (1.748s)
2022-09-27T15:29:47.9455198Z 
2022-09-27T15:29:47.9455681Z ----------------------------------------------------------------------
2022-09-27T15:29:47.9456123Z Ran 1 test in 1.749s
2022-09-27T15:29:47.9456339Z 
2022-09-27T15:29:47.9456436Z OK
2022-09-27T15:29:47.9456538Z 
2022-09-27T15:29:47.9456626Z Generating XML reports...
2022-09-27T15:29:47.9491017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152946.xml
2022-09-27T15:29:49.2726455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:49.2726848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:49.2729848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:49.2730386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:49.4132402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:49.4141354Z 
2022-09-27T15:29:49.4141721Z Running tests...
2022-09-27T15:29:49.4142369Z ----------------------------------------------------------------------
2022-09-27T15:29:49.6453561Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1444
2022-09-27T15:29:49.6478148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1445
2022-09-27T15:29:50.7190408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:50.7191107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:50.7192243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:50.7192962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:50.7306848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:50.7307578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:50.7310833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:50.7311470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:50.8543268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:50.8625898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:51.1517478Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:29:51.1517733Z 
2022-09-27T15:29:51.1518092Z ----------------------------------------------------------------------
2022-09-27T15:29:51.1518465Z Ran 1 test in 1.737s
2022-09-27T15:29:51.1518601Z 
2022-09-27T15:29:51.1518680Z OK (skipped=1)
2022-09-27T15:29:51.1518803Z 
2022-09-27T15:29:51.1518906Z Generating XML reports...
2022-09-27T15:29:51.1550650Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152949.xml
2022-09-27T15:29:52.5014906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:52.5015374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:52.5016248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:52.5016949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:52.6361751Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:52.6371286Z 
2022-09-27T15:29:52.6371727Z Running tests...
2022-09-27T15:29:52.6372139Z ----------------------------------------------------------------------
2022-09-27T15:29:52.6378337Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T15:29:52.8701820Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1477
2022-09-27T15:29:52.8727305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1478
2022-09-27T15:29:53.9188634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:53.9189303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:53.9190385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:53.9190955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:53.9352013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:53.9352683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:53.9353815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:53.9354478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:54.0510911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:54.0675654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:54.3766641Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:29:54.3766898Z 
2022-09-27T15:29:54.3767417Z ----------------------------------------------------------------------
2022-09-27T15:29:54.3767850Z Ran 1 test in 1.739s
2022-09-27T15:29:54.3767988Z 
2022-09-27T15:29:54.3768062Z OK (skipped=1)
2022-09-27T15:29:54.3768170Z 
2022-09-27T15:29:54.3768256Z Generating XML reports...
2022-09-27T15:29:54.3800573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152952.xml
2022-09-27T15:29:55.7162547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:55.7163040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:55.7164291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:55.7164880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:55.8505570Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:55.8514856Z 
2022-09-27T15:29:55.8514996Z Running tests...
2022-09-27T15:29:55.8515638Z ----------------------------------------------------------------------
2022-09-27T15:29:55.8522121Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:29:56.0821058Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1510
2022-09-27T15:29:56.0847766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1511
2022-09-27T15:29:57.1648216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:57.1648639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:57.1650854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:57.1651230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:57.1845430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:57.1845967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:57.1848081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:57.1848439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:57.2978112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:29:57.3199568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:29:57.5884716Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:29:57.5884999Z 
2022-09-27T15:29:57.5885469Z ----------------------------------------------------------------------
2022-09-27T15:29:57.5885817Z Ran 1 test in 1.737s
2022-09-27T15:29:57.5885933Z 
2022-09-27T15:29:57.5886007Z OK (skipped=1)
2022-09-27T15:29:57.5886123Z 
2022-09-27T15:29:57.5886209Z Generating XML reports...
2022-09-27T15:29:57.5918940Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152955.xml
2022-09-27T15:29:58.9329252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:29:58.9329812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:29:58.9331845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:29:58.9332426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:29:59.0685905Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:29:59.0695523Z 
2022-09-27T15:29:59.0695993Z Running tests...
2022-09-27T15:29:59.0696427Z ----------------------------------------------------------------------
2022-09-27T15:29:59.0704586Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:29:59.3015480Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1543
2022-09-27T15:29:59.3041335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1544
2022-09-27T15:30:00.3523942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:00.3524521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:00.3526763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:00.3527370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:00.4060140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:00.4060779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:00.4062348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:00.4062949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:00.4845329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:00.5427945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:00.8080004Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:30:00.8080269Z 
2022-09-27T15:30:00.8080791Z ----------------------------------------------------------------------
2022-09-27T15:30:00.8081122Z Ran 1 test in 1.738s
2022-09-27T15:30:00.8081445Z 
2022-09-27T15:30:00.8081522Z OK (skipped=1)
2022-09-27T15:30:00.8081630Z 
2022-09-27T15:30:00.8081718Z Generating XML reports...
2022-09-27T15:30:00.8113419Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152959.xml
2022-09-27T15:30:02.1653667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:02.1654121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:02.1655897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:02.1656269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:02.2997276Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:02.3007511Z 
2022-09-27T15:30:02.3007949Z Running tests...
2022-09-27T15:30:02.3008532Z ----------------------------------------------------------------------
2022-09-27T15:30:02.3016111Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:30:02.5306716Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1576
2022-09-27T15:30:02.5331873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1577
2022-09-27T15:30:03.6178656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:03.6179023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:03.6180574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:03.6181117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:03.6314149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:03.6314644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:03.6317030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:03.6317406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:03.7510599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:03.7654485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:04.0369946Z skip: Need at least 2 CUDA devices (1.736s)
2022-09-27T15:30:04.0370130Z 
2022-09-27T15:30:04.0370578Z ----------------------------------------------------------------------
2022-09-27T15:30:04.0371007Z Ran 1 test in 1.736s
2022-09-27T15:30:04.0371213Z 
2022-09-27T15:30:04.0371331Z OK (skipped=1)
2022-09-27T15:30:04.0371489Z 
2022-09-27T15:30:04.0371589Z Generating XML reports...
2022-09-27T15:30:04.0404056Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153002.xml
2022-09-27T15:30:05.3839440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:05.3840022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:05.3841748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:05.3842308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:05.5191633Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:05.5201570Z 
2022-09-27T15:30:05.5201824Z Running tests...
2022-09-27T15:30:05.5202462Z ----------------------------------------------------------------------
2022-09-27T15:30:05.5209674Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:30:05.7554588Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1609
2022-09-27T15:30:05.7579930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1610
2022-09-27T15:30:06.8424809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:06.8426131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:06.8426979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:06.8427673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:06.8470218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:06.8470864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:06.8474412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:06.8475051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:06.9751612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:06.9792907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:07.2617075Z skip: Need at least 2 CUDA devices (1.741s)
2022-09-27T15:30:07.2617319Z 
2022-09-27T15:30:07.2617709Z ----------------------------------------------------------------------
2022-09-27T15:30:07.2617996Z Ran 1 test in 1.741s
2022-09-27T15:30:07.2618113Z 
2022-09-27T15:30:07.2618186Z OK (skipped=1)
2022-09-27T15:30:07.2618282Z 
2022-09-27T15:30:07.2618368Z Generating XML reports...
2022-09-27T15:30:07.2651433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153005.xml
2022-09-27T15:30:08.6050370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:08.6050821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:08.6052147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:08.6052685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:08.7398229Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:08.7407388Z 
2022-09-27T15:30:08.7407517Z Running tests...
2022-09-27T15:30:08.7408085Z ----------------------------------------------------------------------
2022-09-27T15:30:08.7414464Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:30:08.9731060Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1642
2022-09-27T15:30:08.9757472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1643
2022-09-27T15:30:10.0292128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:10.0292687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:10.0294144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:10.0294645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:10.0490234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:10.0490798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:10.0493120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:10.0493713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:10.1616377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:10.1841973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:10.4794785Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:30:10.4795050Z 
2022-09-27T15:30:10.4795478Z ----------------------------------------------------------------------
2022-09-27T15:30:10.4795721Z Ran 1 test in 1.739s
2022-09-27T15:30:10.4795838Z 
2022-09-27T15:30:10.4795945Z OK (skipped=1)
2022-09-27T15:30:10.4796090Z 
2022-09-27T15:30:10.4796179Z Generating XML reports...
2022-09-27T15:30:10.4828229Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153008.xml
2022-09-27T15:30:11.8263779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:11.8264355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:11.8266165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:11.8266579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:11.9614272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:11.9623590Z 
2022-09-27T15:30:11.9623734Z Running tests...
2022-09-27T15:30:11.9624352Z ----------------------------------------------------------------------
2022-09-27T15:30:11.9633544Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:30:12.1941635Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1675
2022-09-27T15:30:12.1965885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1676
2022-09-27T15:30:13.2793139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:13.2793527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:13.2795637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:13.2796001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:13.2904039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:13.2904409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:13.2907768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:13.2908125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:13.4137414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:13.4224356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:13.7003347Z skip: Need at least 2 CUDA devices (1.738s)
2022-09-27T15:30:13.7016721Z 
2022-09-27T15:30:13.7017146Z ----------------------------------------------------------------------
2022-09-27T15:30:13.7017518Z Ran 1 test in 1.738s
2022-09-27T15:30:13.7017719Z 
2022-09-27T15:30:13.7017856Z OK (skipped=1)
2022-09-27T15:30:13.7018053Z 
2022-09-27T15:30:13.7019919Z Generating XML reports...
2022-09-27T15:30:13.7039220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153011.xml
2022-09-27T15:30:15.0483386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:15.0483854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:15.0485059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:15.0485517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:15.1828181Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:15.1836564Z 
2022-09-27T15:30:15.1836701Z Running tests...
2022-09-27T15:30:15.1837080Z ----------------------------------------------------------------------
2022-09-27T15:30:15.1846996Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:30:15.4162827Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1708
2022-09-27T15:30:15.4188557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1709
2022-09-27T15:30:16.4925702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:16.4926268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:16.4927222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:16.4927702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:16.5194376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:16.5194998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:16.5196071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:16.5196670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:16.6251649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:16.6520009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:16.9226354Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:30:16.9226614Z 
2022-09-27T15:30:16.9227149Z ----------------------------------------------------------------------
2022-09-27T15:30:16.9227497Z Ran 1 test in 1.739s
2022-09-27T15:30:16.9227612Z 
2022-09-27T15:30:16.9227688Z OK (skipped=1)
2022-09-27T15:30:16.9227798Z 
2022-09-27T15:30:16.9227871Z Generating XML reports...
2022-09-27T15:30:16.9259946Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153015.xml
2022-09-27T15:30:18.3096283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:18.3096710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:18.3098567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:18.3099152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:18.4482363Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:18.4492393Z 
2022-09-27T15:30:18.4492525Z Running tests...
2022-09-27T15:30:18.4493233Z ----------------------------------------------------------------------
2022-09-27T15:30:18.4499565Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:30:18.6863765Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1741
2022-09-27T15:30:18.6890041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1742
2022-09-27T15:30:19.7312490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:19.7312952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:19.7314040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:19.7314545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:19.7587832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:19.7588257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:19.7590574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:19.7590976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:19.8638424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:19.8941903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:20.1929280Z skip: Need at least 2 CUDA devices (1.743s)
2022-09-27T15:30:20.1929627Z 
2022-09-27T15:30:20.1930018Z ----------------------------------------------------------------------
2022-09-27T15:30:20.1930323Z Ran 1 test in 1.743s
2022-09-27T15:30:20.1930443Z 
2022-09-27T15:30:20.1930533Z OK (skipped=1)
2022-09-27T15:30:20.1930658Z 
2022-09-27T15:30:20.1930760Z Generating XML reports...
2022-09-27T15:30:20.1962609Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153018.xml
2022-09-27T15:30:21.5568273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:21.5568713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:21.5570007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:21.5570563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:21.6918478Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:21.6927779Z 
2022-09-27T15:30:21.6927946Z Running tests...
2022-09-27T15:30:21.6928432Z ----------------------------------------------------------------------
2022-09-27T15:30:21.6938145Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:30:21.9260485Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1774
2022-09-27T15:30:21.9286189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1775
2022-09-27T15:30:23.0101660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:23.0102113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:23.0103499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:23.0104061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:23.0755958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:23.0756351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:23.0758874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:23.0759263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:23.1450756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:23.2137459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:23.4322391Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:30:23.4322632Z 
2022-09-27T15:30:23.4323369Z ----------------------------------------------------------------------
2022-09-27T15:30:23.4323807Z Ran 1 test in 1.739s
2022-09-27T15:30:23.4323935Z 
2022-09-27T15:30:23.4324008Z OK (skipped=1)
2022-09-27T15:30:23.4324114Z 
2022-09-27T15:30:23.4324200Z Generating XML reports...
2022-09-27T15:30:23.4358522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153021.xml
2022-09-27T15:30:24.7556271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:24.7556681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:24.7558017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:24.7558664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:24.8901816Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:24.8910861Z 
2022-09-27T15:30:24.8910996Z Running tests...
2022-09-27T15:30:24.8911681Z ----------------------------------------------------------------------
2022-09-27T15:30:24.8920946Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:30:25.1248012Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1807
2022-09-27T15:30:25.1273472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1808
2022-09-27T15:30:26.2020756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:26.2021270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:26.2024053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:26.2024699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:26.2054661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:26.2055211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:26.2058248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:26.2058968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:26.3349836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:26.3373706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:26.6312377Z skip: Need at least 2 CUDA devices (1.740s)
2022-09-27T15:30:26.6312648Z 
2022-09-27T15:30:26.6313081Z ----------------------------------------------------------------------
2022-09-27T15:30:26.6313510Z Ran 1 test in 1.740s
2022-09-27T15:30:26.6313701Z 
2022-09-27T15:30:26.6313822Z OK (skipped=1)
2022-09-27T15:30:26.6313987Z 
2022-09-27T15:30:26.6314119Z Generating XML reports...
2022-09-27T15:30:26.6346394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153024.xml
2022-09-27T15:30:27.9778169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:27.9778577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:27.9780493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:27.9781280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:28.1121089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:28.1130527Z 
2022-09-27T15:30:28.1130966Z Running tests...
2022-09-27T15:30:28.1131569Z ----------------------------------------------------------------------
2022-09-27T15:30:28.1141474Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:30:28.3507023Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1840
2022-09-27T15:30:28.3532931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1841
2022-09-27T15:30:29.4003629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:29.4004242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:29.4006232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:29.4006804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:29.4260590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:29.4261278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:29.4263141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:29.4263764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:29.5329785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:29.5606720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:29.8571160Z skip: Need at least 2 CUDA devices (1.744s)
2022-09-27T15:30:29.8571419Z 
2022-09-27T15:30:29.8571877Z ----------------------------------------------------------------------
2022-09-27T15:30:29.8572275Z Ran 1 test in 1.744s
2022-09-27T15:30:29.8572434Z 
2022-09-27T15:30:29.8572546Z OK (skipped=1)
2022-09-27T15:30:29.8572725Z 
2022-09-27T15:30:29.8572861Z Generating XML reports...
2022-09-27T15:30:29.8605835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153028.xml
2022-09-27T15:30:31.2261894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:31.2262288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:31.2264288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:31.2264655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:31.3732263Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:31.3741791Z 
2022-09-27T15:30:31.3741925Z Running tests...
2022-09-27T15:30:31.3742332Z ----------------------------------------------------------------------
2022-09-27T15:30:31.3754275Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:30:31.6117270Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1873
2022-09-27T15:30:31.6142779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1874
2022-09-27T15:30:32.6784848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:32.6785314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:32.6786305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:32.6787058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:32.6920419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:32.6921046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:32.6923791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:32.6924361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:32.8104089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:32.8233785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:33.1180523Z skip: Need at least 2 CUDA devices (1.744s)
2022-09-27T15:30:33.1180837Z 
2022-09-27T15:30:33.1181377Z ----------------------------------------------------------------------
2022-09-27T15:30:33.1181760Z Ran 1 test in 1.744s
2022-09-27T15:30:33.1181890Z 
2022-09-27T15:30:33.1181977Z OK (skipped=1)
2022-09-27T15:30:33.1182100Z 
2022-09-27T15:30:33.1182201Z Generating XML reports...
2022-09-27T15:30:33.1214468Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153031.xml
2022-09-27T15:30:34.4740357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:34.4740733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:34.4742728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:34.4743071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:34.6085521Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:34.6094731Z 
2022-09-27T15:30:34.6094855Z Running tests...
2022-09-27T15:30:34.6095453Z ----------------------------------------------------------------------
2022-09-27T15:30:34.6102716Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-09-27T15:30:34.8402519Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1906
2022-09-27T15:30:34.8427894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1907
2022-09-27T15:30:35.9254119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:35.9254524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:35.9255994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:35.9256503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:35.9281079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:35.9281560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:35.9284377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:35.9284868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:36.0573254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:36.0594447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:36.0844450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_38j5tx9
2022-09-27T15:30:36.0845118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rqo01ut
2022-09-27T15:30:36.0846463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_38j5tx9/_remote_module_non_scriptable.py
2022-09-27T15:30:36.0847116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rqo01ut/_remote_module_non_scriptable.py
2022-09-27T15:30:36.3464362Z ok (1.737s)
2022-09-27T15:30:36.3464950Z 
2022-09-27T15:30:36.3465363Z ----------------------------------------------------------------------
2022-09-27T15:30:36.3465700Z Ran 1 test in 1.737s
2022-09-27T15:30:36.3465817Z 
2022-09-27T15:30:36.3465877Z OK
2022-09-27T15:30:36.3465967Z 
2022-09-27T15:30:36.3466065Z Generating XML reports...
2022-09-27T15:30:36.3497198Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153034.xml
2022-09-27T15:30:37.6741366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:37.6741954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:37.6743365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:37.6744176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:37.8085665Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:37.8095718Z 
2022-09-27T15:30:37.8096183Z Running tests...
2022-09-27T15:30:37.8096618Z ----------------------------------------------------------------------
2022-09-27T15:30:37.8102904Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-09-27T15:30:38.0405287Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1945
2022-09-27T15:30:38.0430754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1946
2022-09-27T15:30:39.1055306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:39.1055776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:39.1057314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:39.1057743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:39.1124780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:39.1125134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:39.1128041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:39.1128386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:39.2376287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:39.2442210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:39.5467510Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:30:39.5467782Z 
2022-09-27T15:30:39.5468239Z ----------------------------------------------------------------------
2022-09-27T15:30:39.5468662Z Ran 1 test in 1.737s
2022-09-27T15:30:39.5468842Z 
2022-09-27T15:30:39.5468959Z OK (skipped=1)
2022-09-27T15:30:39.5469123Z 
2022-09-27T15:30:39.5469265Z Generating XML reports...
2022-09-27T15:30:39.5501046Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153037.xml
2022-09-27T15:30:40.8967014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:40.8967455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:40.8969354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:40.8969970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:41.0321391Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:41.0330839Z 
2022-09-27T15:30:41.0331004Z Running tests...
2022-09-27T15:30:41.0332203Z ----------------------------------------------------------------------
2022-09-27T15:30:41.0340502Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-09-27T15:30:41.2719911Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1978
2022-09-27T15:30:41.2746692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1979
2022-09-27T15:30:42.3471061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:42.3471780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:42.3472522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:42.3472898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:42.3792391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:42.3792771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:42.3795387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:42.3795764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:42.4796774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:42.5103496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:42.5366124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1t4peix
2022-09-27T15:30:42.5366657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizjywe_p
2022-09-27T15:30:42.5367477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1t4peix/_remote_module_non_scriptable.py
2022-09-27T15:30:42.5367947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizjywe_p/_remote_module_non_scriptable.py
2022-09-27T15:30:42.8784558Z ok (1.845s)
2022-09-27T15:30:42.8784782Z 
2022-09-27T15:30:42.8785146Z ----------------------------------------------------------------------
2022-09-27T15:30:42.8785471Z Ran 1 test in 1.845s
2022-09-27T15:30:42.8785600Z 
2022-09-27T15:30:42.8785662Z OK
2022-09-27T15:30:42.8785754Z 
2022-09-27T15:30:42.8785852Z Generating XML reports...
2022-09-27T15:30:42.8817955Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153041.xml
2022-09-27T15:30:44.2342597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:44.2343187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:44.2344967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:44.2345611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:44.3688629Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:44.3697642Z 
2022-09-27T15:30:44.3697882Z Running tests...
2022-09-27T15:30:44.3698349Z ----------------------------------------------------------------------
2022-09-27T15:30:44.3708990Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:30:44.6017181Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2017
2022-09-27T15:30:44.6042659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2018
2022-09-27T15:30:45.6838728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:45.6839350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:45.6841651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:45.6842135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:45.7027196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:45.7027579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:45.7030544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:45.7030947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:45.8222698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:45.8384905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:45.8598819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti91ym0p
2022-09-27T15:30:45.8599457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3q0ym17
2022-09-27T15:30:45.8600428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti91ym0p/_remote_module_non_scriptable.py
2022-09-27T15:30:45.8601039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3q0ym17/_remote_module_non_scriptable.py
2022-09-27T15:30:46.2081056Z ok (1.838s)
2022-09-27T15:30:46.2081273Z 
2022-09-27T15:30:46.2081757Z ----------------------------------------------------------------------
2022-09-27T15:30:46.2082190Z Ran 1 test in 1.838s
2022-09-27T15:30:46.2082366Z 
2022-09-27T15:30:46.2082432Z OK
2022-09-27T15:30:46.2082525Z 
2022-09-27T15:30:46.2082619Z Generating XML reports...
2022-09-27T15:30:46.2114638Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153044.xml
2022-09-27T15:30:47.5300039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:47.5300579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:47.5301414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:47.5301771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:47.6642168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:47.6652215Z 
2022-09-27T15:30:47.6652502Z Running tests...
2022-09-27T15:30:47.6653191Z ----------------------------------------------------------------------
2022-09-27T15:30:47.6661784Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-09-27T15:30:47.9018851Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2062
2022-09-27T15:30:47.9044160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2063
2022-09-27T15:30:48.9563764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:48.9564393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:48.9565146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:48.9565702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:48.9975955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:48.9976610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:48.9977931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:48.9978750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:49.0893682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:49.1322256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:49.1490815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4t7k95p
2022-09-27T15:30:49.1491479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_80ba4p
2022-09-27T15:30:49.1492602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4t7k95p/_remote_module_non_scriptable.py
2022-09-27T15:30:49.1493246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_80ba4p/_remote_module_non_scriptable.py
2022-09-27T15:30:49.4081532Z ok (1.743s)
2022-09-27T15:30:49.4081792Z 
2022-09-27T15:30:49.4082341Z ----------------------------------------------------------------------
2022-09-27T15:30:49.4082643Z Ran 1 test in 1.743s
2022-09-27T15:30:49.4082749Z 
2022-09-27T15:30:49.4082812Z OK
2022-09-27T15:30:49.4082902Z 
2022-09-27T15:30:49.4083000Z Generating XML reports...
2022-09-27T15:30:49.4115121Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153047.xml
2022-09-27T15:30:50.7284447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:50.7284932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:50.7286394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:50.7286934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:50.8643134Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:50.8652848Z 
2022-09-27T15:30:50.8653331Z Running tests...
2022-09-27T15:30:50.8653800Z ----------------------------------------------------------------------
2022-09-27T15:30:50.8664401Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-09-27T15:30:51.1003137Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2101
2022-09-27T15:30:51.1028251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2102
2022-09-27T15:30:52.1619717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:52.1620172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:52.1622643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:52.1623039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:52.1895551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:52.1895888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:52.1898711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:52.1899322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:52.2944515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:52.3234622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:52.3503305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdizn_ka
2022-09-27T15:30:52.3504184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ej10qht
2022-09-27T15:30:52.3505225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdizn_ka/_remote_module_non_scriptable.py
2022-09-27T15:30:52.3505976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ej10qht/_remote_module_non_scriptable.py
2022-09-27T15:30:52.6069680Z ok (1.741s)
2022-09-27T15:30:52.6069915Z 
2022-09-27T15:30:52.6070276Z ----------------------------------------------------------------------
2022-09-27T15:30:52.6070554Z Ran 1 test in 1.742s
2022-09-27T15:30:52.6070671Z 
2022-09-27T15:30:52.6070734Z OK
2022-09-27T15:30:52.6070813Z 
2022-09-27T15:30:52.6070909Z Generating XML reports...
2022-09-27T15:30:52.6102498Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153050.xml
2022-09-27T15:30:53.9359093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:53.9359789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:53.9360939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:53.9361570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:54.0704415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:54.0715176Z 
2022-09-27T15:30:54.0715620Z Running tests...
2022-09-27T15:30:54.0716025Z ----------------------------------------------------------------------
2022-09-27T15:30:54.0730797Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-09-27T15:30:54.3071444Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2140
2022-09-27T15:30:54.3096300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2141
2022-09-27T15:30:55.3853311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:55.3854119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:55.3855548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:55.3855946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:55.3895009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:55.3895488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:55.3898333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:55.3898890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:55.5178592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:55.5207960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:55.8133012Z skip: Need at least 2 CUDA devices (1.741s)
2022-09-27T15:30:55.8133282Z 
2022-09-27T15:30:55.8133755Z ----------------------------------------------------------------------
2022-09-27T15:30:55.8134113Z Ran 1 test in 1.742s
2022-09-27T15:30:55.8134298Z 
2022-09-27T15:30:55.8134416Z OK (skipped=1)
2022-09-27T15:30:55.8134594Z 
2022-09-27T15:30:55.8134725Z Generating XML reports...
2022-09-27T15:30:55.8167617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153054.xml
2022-09-27T15:30:57.1569926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:57.1570345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:57.1571372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:57.1571893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:57.2917673Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:30:57.2926954Z 
2022-09-27T15:30:57.2927067Z Running tests...
2022-09-27T15:30:57.2927486Z ----------------------------------------------------------------------
2022-09-27T15:30:57.5263775Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2173
2022-09-27T15:30:57.5289639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2174
2022-09-27T15:30:58.5771339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:58.5772333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:58.5772835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:58.5773211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:58.6189818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:30:58.6190195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:30:58.6193009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:30:58.6193388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:30:58.7090024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:30:58.7550958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:30:59.0325859Z skip: Need at least 2 CUDA devices (1.740s)
2022-09-27T15:30:59.0326197Z 
2022-09-27T15:30:59.0326760Z ----------------------------------------------------------------------
2022-09-27T15:30:59.0327113Z Ran 1 test in 1.740s
2022-09-27T15:30:59.0327232Z 
2022-09-27T15:30:59.0327322Z OK (skipped=1)
2022-09-27T15:30:59.0327447Z 
2022-09-27T15:30:59.0327549Z Generating XML reports...
2022-09-27T15:30:59.0358625Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153057.xml
2022-09-27T15:31:00.3740917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:00.3741312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:00.3743293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:00.3743806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:00.5079867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:00.5089342Z 
2022-09-27T15:31:00.5089486Z Running tests...
2022-09-27T15:31:00.5090058Z ----------------------------------------------------------------------
2022-09-27T15:31:00.7402001Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2206
2022-09-27T15:31:00.7427220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2207
2022-09-27T15:31:01.8237252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:01.8237783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:01.8238904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:01.8239435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:01.8468618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:01.8469262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:01.8471932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:01.8472536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:01.9560040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:01.9821851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:02.2466165Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:31:02.2466532Z 
2022-09-27T15:31:02.2467179Z ----------------------------------------------------------------------
2022-09-27T15:31:02.2467714Z Ran 1 test in 1.738s
2022-09-27T15:31:02.2467890Z 
2022-09-27T15:31:02.2467973Z OK (skipped=1)
2022-09-27T15:31:02.2468084Z 
2022-09-27T15:31:02.2468171Z Generating XML reports...
2022-09-27T15:31:02.2499158Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153100.xml
2022-09-27T15:31:03.5883643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:03.5885545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:03.5886366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:03.5886969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:03.7229864Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:03.7239675Z 
2022-09-27T15:31:03.7239975Z Running tests...
2022-09-27T15:31:03.7240622Z ----------------------------------------------------------------------
2022-09-27T15:31:03.9573267Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2239
2022-09-27T15:31:03.9599865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2240
2022-09-27T15:31:05.0093126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:05.0093515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:05.0095902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:05.0096404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:05.0316146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:05.0316532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:05.0318762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:05.0319331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:05.1419054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:05.1674396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:05.4637820Z skip: Need at least 2 CUDA devices (1.740s)
2022-09-27T15:31:05.4638114Z 
2022-09-27T15:31:05.4638574Z ----------------------------------------------------------------------
2022-09-27T15:31:05.4638814Z Ran 1 test in 1.740s
2022-09-27T15:31:05.4639109Z 
2022-09-27T15:31:05.4639186Z OK (skipped=1)
2022-09-27T15:31:05.4639296Z 
2022-09-27T15:31:05.4639385Z Generating XML reports...
2022-09-27T15:31:05.4671356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153103.xml
2022-09-27T15:31:06.8033605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:06.8034002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:06.8035683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:06.8036192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:06.9375081Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:06.9384374Z 
2022-09-27T15:31:06.9384510Z Running tests...
2022-09-27T15:31:06.9385122Z ----------------------------------------------------------------------
2022-09-27T15:31:07.1708985Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2272
2022-09-27T15:31:07.1734973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2273
2022-09-27T15:31:08.2700940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:08.2701303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:08.2703756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:08.2704236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:08.3126401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:08.3126825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:08.3129051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:08.3129450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:08.4026225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:08.4480005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:08.6773816Z skip: Need at least 2 CUDA devices (1.739s)
2022-09-27T15:31:08.6774116Z 
2022-09-27T15:31:08.6774661Z ----------------------------------------------------------------------
2022-09-27T15:31:08.6775006Z Ran 1 test in 1.739s
2022-09-27T15:31:08.6775109Z 
2022-09-27T15:31:08.6775186Z OK (skipped=1)
2022-09-27T15:31:08.6775294Z 
2022-09-27T15:31:08.6775382Z Generating XML reports...
2022-09-27T15:31:08.6807458Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153106.xml
2022-09-27T15:31:09.9970981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:09.9971513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:09.9972916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:09.9973458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:10.1311676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:10.1320895Z 
2022-09-27T15:31:10.1321024Z Running tests...
2022-09-27T15:31:10.1321847Z ----------------------------------------------------------------------
2022-09-27T15:31:10.3667272Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2305
2022-09-27T15:31:10.3691594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2306
2022-09-27T15:31:11.4551008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:11.4551436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:11.4553397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:11.4553925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:11.4855007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:11.4855446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:11.4857952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:11.4858490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:11.5868756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:11.6201102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:11.8727840Z skip: Need at least 2 CUDA devices (1.740s)
2022-09-27T15:31:11.8728045Z 
2022-09-27T15:31:11.8728434Z ----------------------------------------------------------------------
2022-09-27T15:31:11.8728692Z Ran 1 test in 1.741s
2022-09-27T15:31:11.8728837Z 
2022-09-27T15:31:11.8728941Z OK (skipped=1)
2022-09-27T15:31:11.8729055Z 
2022-09-27T15:31:11.8729142Z Generating XML reports...
2022-09-27T15:31:11.8761556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153110.xml
2022-09-27T15:31:13.1859328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:13.1859768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:13.1861550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:13.1861911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:13.3202604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:13.3212086Z 
2022-09-27T15:31:13.3212165Z Running tests...
2022-09-27T15:31:13.3213283Z ----------------------------------------------------------------------
2022-09-27T15:31:13.5543343Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2338
2022-09-27T15:31:13.5569141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2339
2022-09-27T15:31:14.6188003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:14.6188368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:14.6190499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:14.6190887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:14.6781173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:14.6781534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:14.6784001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:14.6784378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:14.7509440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:14.8143548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:15.0608914Z skip: Need at least 4 CUDA devices (1.739s)
2022-09-27T15:31:15.0609211Z 
2022-09-27T15:31:15.0609644Z ----------------------------------------------------------------------
2022-09-27T15:31:15.0609954Z Ran 1 test in 1.740s
2022-09-27T15:31:15.0610074Z 
2022-09-27T15:31:15.0610152Z OK (skipped=1)
2022-09-27T15:31:15.0610263Z 
2022-09-27T15:31:15.0610348Z Generating XML reports...
2022-09-27T15:31:15.0642315Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153113.xml
2022-09-27T15:31:16.3964581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:16.3965019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:16.3967313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:16.3967715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:16.5410507Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:16.5419723Z 
2022-09-27T15:31:16.5420166Z Running tests...
2022-09-27T15:31:16.5420549Z ----------------------------------------------------------------------
2022-09-27T15:31:16.7751081Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2371
2022-09-27T15:31:16.7777678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2372
2022-09-27T15:31:17.8563860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:17.8564311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:17.8565616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:17.8566183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:17.8969347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:17.8969865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:17.8970949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:17.8971286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:17.9890600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:18.0284559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:18.2814526Z skip: Need at least 8 CUDA devices (1.739s)
2022-09-27T15:31:18.2814783Z 
2022-09-27T15:31:18.2815164Z ----------------------------------------------------------------------
2022-09-27T15:31:18.2815459Z Ran 1 test in 1.739s
2022-09-27T15:31:18.2815654Z 
2022-09-27T15:31:18.2815794Z OK (skipped=1)
2022-09-27T15:31:18.2816000Z 
2022-09-27T15:31:18.2816160Z Generating XML reports...
2022-09-27T15:31:18.2851628Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153116.xml
2022-09-27T15:31:19.6298819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:19.6299201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:19.6301134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:19.6301493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:19.7676956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:19.7686782Z 
2022-09-27T15:31:19.7687008Z Running tests...
2022-09-27T15:31:19.7687771Z ----------------------------------------------------------------------
2022-09-27T15:31:20.0038177Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2404
2022-09-27T15:31:20.0063554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2405
2022-09-27T15:31:21.0799600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:21.0800111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:21.0801250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:21.0801633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:21.0899977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:21.0900317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:21.0903282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:21.0903650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:21.2121677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:21.2214762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:21.2387229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sl46f6_
2022-09-27T15:31:21.2387696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaa5jxr72
2022-09-27T15:31:21.2389495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sl46f6_/_remote_module_non_scriptable.py
2022-09-27T15:31:21.2389898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaa5jxr72/_remote_module_non_scriptable.py
2022-09-27T15:31:21.2535051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:21.2535471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:21.5100526Z ok (1.741s)
2022-09-27T15:31:21.5100765Z 
2022-09-27T15:31:21.5101293Z ----------------------------------------------------------------------
2022-09-27T15:31:21.5101673Z Ran 1 test in 1.741s
2022-09-27T15:31:21.5101793Z 
2022-09-27T15:31:21.5101862Z OK
2022-09-27T15:31:21.5101954Z 
2022-09-27T15:31:21.5102050Z Generating XML reports...
2022-09-27T15:31:21.5136192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153119.xml
2022-09-27T15:31:22.8370606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:22.8371007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:22.8372916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:22.8373443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:22.9717651Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:22.9727226Z 
2022-09-27T15:31:22.9727354Z Running tests...
2022-09-27T15:31:22.9727941Z ----------------------------------------------------------------------
2022-09-27T15:31:23.2118488Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2443
2022-09-27T15:31:23.2143807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2444
2022-09-27T15:31:24.3018152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:24.3019109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:24.3020271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:24.3020858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:24.3311771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:24.3312428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:24.3314906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:24.3315548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:24.4343452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:24.4676486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:24.4951242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh0jx5kh
2022-09-27T15:31:24.4951953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiww4ixco
2022-09-27T15:31:24.4952731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh0jx5kh/_remote_module_non_scriptable.py
2022-09-27T15:31:24.4954043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiww4ixco/_remote_module_non_scriptable.py
2022-09-27T15:31:24.5106868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:24.5107240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:24.8181565Z ok (1.845s)
2022-09-27T15:31:24.8181828Z 
2022-09-27T15:31:24.8182348Z ----------------------------------------------------------------------
2022-09-27T15:31:24.8182653Z Ran 1 test in 1.845s
2022-09-27T15:31:24.8182782Z 
2022-09-27T15:31:24.8182845Z OK
2022-09-27T15:31:24.8182938Z 
2022-09-27T15:31:24.8183030Z Generating XML reports...
2022-09-27T15:31:24.8214492Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153122.xml
2022-09-27T15:31:26.1462146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:26.1462742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:26.1464582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:26.1465156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:26.2807281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:26.2816471Z 
2022-09-27T15:31:26.2816612Z Running tests...
2022-09-27T15:31:26.2817218Z ----------------------------------------------------------------------
2022-09-27T15:31:26.2832203Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-09-27T15:31:26.5162522Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2482
2022-09-27T15:31:26.5188381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2483
2022-09-27T15:31:27.6062504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:27.6063167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:27.6064682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:27.6065588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:27.6090893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:27.6091697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:27.6094297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:27.6094917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:27.7390982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:27.7408155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:27.7663203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjijf5hrg
2022-09-27T15:31:27.7664072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jdqz5dq
2022-09-27T15:31:27.7665042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjijf5hrg/_remote_module_non_scriptable.py
2022-09-27T15:31:27.7666239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jdqz5dq/_remote_module_non_scriptable.py
2022-09-27T15:31:27.7830230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:27.7830747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:31:28.1228796Z ok (1.841s)
2022-09-27T15:31:28.1228981Z 
2022-09-27T15:31:28.1229722Z ----------------------------------------------------------------------
2022-09-27T15:31:28.1230014Z Ran 1 test in 1.841s
2022-09-27T15:31:28.1230134Z 
2022-09-27T15:31:28.1230205Z OK
2022-09-27T15:31:28.1230299Z 
2022-09-27T15:31:28.1230391Z Generating XML reports...
2022-09-27T15:31:28.1262833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153126.xml
2022-09-27T15:31:29.4529494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:29.4529895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:29.4531728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:29.4532130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:29.5868916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:29.5878186Z 
2022-09-27T15:31:29.5878284Z Running tests...
2022-09-27T15:31:29.5878830Z ----------------------------------------------------------------------
2022-09-27T15:31:29.5894030Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-09-27T15:31:29.8205613Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2527
2022-09-27T15:31:29.8230824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2528
2022-09-27T15:31:30.8938291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:30.8938874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:30.8940023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:30.8940599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:30.8968735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:30.8969376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:30.8972879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:30.8973257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:31.0274196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:31.0286225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:31.0546295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq1197c3
2022-09-27T15:31:31.0546898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f6hsjq6
2022-09-27T15:31:31.0548096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq1197c3/_remote_module_non_scriptable.py
2022-09-27T15:31:31.0548693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f6hsjq6/_remote_module_non_scriptable.py
2022-09-27T15:31:31.3268778Z ok (1.739s)
2022-09-27T15:31:31.3268993Z 
2022-09-27T15:31:31.3269504Z ----------------------------------------------------------------------
2022-09-27T15:31:31.3269880Z Ran 1 test in 1.739s
2022-09-27T15:31:31.3269997Z 
2022-09-27T15:31:31.3270062Z OK
2022-09-27T15:31:31.3270157Z 
2022-09-27T15:31:31.3270250Z Generating XML reports...
2022-09-27T15:31:31.3303810Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153129.xml
2022-09-27T15:31:32.6670959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:32.6671552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:32.6672986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:32.6673572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:32.8017191Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:32.8027755Z 
2022-09-27T15:31:32.8028115Z Running tests...
2022-09-27T15:31:32.8028728Z ----------------------------------------------------------------------
2022-09-27T15:31:33.0386902Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2572
2022-09-27T15:31:33.0412641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2573
2022-09-27T15:31:34.1006500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:34.1006883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:34.1009082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:34.1009435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:34.1133970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:34.1134322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:34.1137088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:34.1137439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:34.2331312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:34.2336315Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2337896Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2339004Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2339830Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2340642Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2341458Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2459719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:34.2464678Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2465864Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2467109Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2468065Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2468894Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.2469705Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:31:34.5450116Z ok (1.742s)
2022-09-27T15:31:34.5450275Z 
2022-09-27T15:31:34.5451101Z ----------------------------------------------------------------------
2022-09-27T15:31:34.5451550Z Ran 1 test in 1.742s
2022-09-27T15:31:34.5451684Z 
2022-09-27T15:31:34.5451767Z OK
2022-09-27T15:31:34.5451950Z 
2022-09-27T15:31:34.5452045Z Generating XML reports...
2022-09-27T15:31:34.5484259Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153132.xml
2022-09-27T15:31:35.8828699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:35.8829173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:35.8830576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:35.8830970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:36.0187834Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:36.0196942Z 
2022-09-27T15:31:36.0197263Z Running tests...
2022-09-27T15:31:36.0197671Z ----------------------------------------------------------------------
2022-09-27T15:31:36.2543022Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2605
2022-09-27T15:31:36.2567729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2606
2022-09-27T15:31:37.3050828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:37.3051234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:37.3053013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:37.3053536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:37.3272266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:37.3272682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:37.3275285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:37.3275652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:37.4371776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:37.4623503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:37.7605585Z skip: Need at least 2 CUDA devices (1.741s)
2022-09-27T15:31:37.7605899Z 
2022-09-27T15:31:37.7606396Z ----------------------------------------------------------------------
2022-09-27T15:31:37.7606651Z Ran 1 test in 1.741s
2022-09-27T15:31:37.7606753Z 
2022-09-27T15:31:37.7606828Z OK (skipped=1)
2022-09-27T15:31:37.7606936Z 
2022-09-27T15:31:37.7607038Z Generating XML reports...
2022-09-27T15:31:37.7639635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153136.xml
2022-09-27T15:31:39.1020139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:39.1020750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:39.1022282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:39.1022859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:39.2371903Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:39.2381473Z 
2022-09-27T15:31:39.2381811Z Running tests...
2022-09-27T15:31:39.2382238Z ----------------------------------------------------------------------
2022-09-27T15:31:39.4701361Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2638
2022-09-27T15:31:39.4726577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2639
2022-09-27T15:31:40.5325316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:40.5325784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:40.5326781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:40.5327370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:40.5678889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:40.5679258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:40.5681713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:40.5682079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:40.6654284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:40.6987937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:40.7264406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53gllzq0
2022-09-27T15:31:40.7265046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wpnwibr
2022-09-27T15:31:40.7265901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53gllzq0/_remote_module_non_scriptable.py
2022-09-27T15:31:40.7266369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wpnwibr/_remote_module_non_scriptable.py
2022-09-27T15:31:41.0767602Z ok (1.838s)
2022-09-27T15:31:41.0767866Z 
2022-09-27T15:31:41.0768422Z ----------------------------------------------------------------------
2022-09-27T15:31:41.0768849Z Ran 1 test in 1.839s
2022-09-27T15:31:41.0769053Z 
2022-09-27T15:31:41.0769172Z OK
2022-09-27T15:31:41.0769339Z 
2022-09-27T15:31:41.0769506Z Generating XML reports...
2022-09-27T15:31:41.0801826Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153139.xml
2022-09-27T15:31:42.4066341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:42.4066726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:42.4068858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:42.4069266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:42.5418993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:42.5429563Z 
2022-09-27T15:31:42.5429768Z Running tests...
2022-09-27T15:31:42.5430379Z ----------------------------------------------------------------------
2022-09-27T15:31:42.7786923Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2683
2022-09-27T15:31:42.7811651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2684
2022-09-27T15:31:43.8361273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:43.8361653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:43.8363691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:43.8364241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:43.8899052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:43.8899417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:43.8901178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:43.8901518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:43.9678657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:44.0242669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:44.0418558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppw4w5vjz
2022-09-27T15:31:44.0420085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayuea4f7
2022-09-27T15:31:44.0420836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppw4w5vjz/_remote_module_non_scriptable.py
2022-09-27T15:31:44.0421985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayuea4f7/_remote_module_non_scriptable.py
2022-09-27T15:31:44.3848665Z ok (1.842s)
2022-09-27T15:31:44.3849134Z 
2022-09-27T15:31:44.3849918Z ----------------------------------------------------------------------
2022-09-27T15:31:44.3850406Z Ran 1 test in 1.842s
2022-09-27T15:31:44.3850539Z 
2022-09-27T15:31:44.3850608Z OK
2022-09-27T15:31:44.3850688Z 
2022-09-27T15:31:44.3850784Z Generating XML reports...
2022-09-27T15:31:44.3882378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153142.xml
2022-09-27T15:31:45.7049303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:45.7049724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:45.7050855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:45.7051211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:45.8385501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:45.8395227Z 
2022-09-27T15:31:45.8395425Z Running tests...
2022-09-27T15:31:45.8395762Z ----------------------------------------------------------------------
2022-09-27T15:31:46.0709605Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2728
2022-09-27T15:31:46.0735625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2729
2022-09-27T15:31:47.1484071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:47.1484536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:47.1485448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:47.1486012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:47.1582710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:47.1583319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:47.1585230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:47.1585935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:47.2823525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:47.2919603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:47.5772962Z skip: Need at least 2 CUDA devices (1.737s)
2022-09-27T15:31:47.5773252Z 
2022-09-27T15:31:47.5773920Z ----------------------------------------------------------------------
2022-09-27T15:31:47.5774227Z Ran 1 test in 1.738s
2022-09-27T15:31:47.5774526Z 
2022-09-27T15:31:47.5774647Z OK (skipped=1)
2022-09-27T15:31:47.5774826Z 
2022-09-27T15:31:47.5774986Z Generating XML reports...
2022-09-27T15:31:47.5806193Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153145.xml
2022-09-27T15:31:48.9587041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:48.9587411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:48.9589703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:48.9590067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:49.0972505Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:49.0983173Z 
2022-09-27T15:31:49.0983429Z Running tests...
2022-09-27T15:31:49.0984210Z ----------------------------------------------------------------------
2022-09-27T15:31:49.3369484Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2761
2022-09-27T15:31:49.3394848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2762
2022-09-27T15:31:50.4043824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:50.4044490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:50.4045748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:50.4046343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:50.4176346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:50.4177040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:50.4179843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:50.4180436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:50.5365378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:50.5531732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:50.8433964Z skip: Need at least 2 CUDA devices (1.745s)
2022-09-27T15:31:50.8434170Z 
2022-09-27T15:31:50.8434488Z ----------------------------------------------------------------------
2022-09-27T15:31:50.8434745Z Ran 1 test in 1.745s
2022-09-27T15:31:50.8434847Z 
2022-09-27T15:31:50.8434934Z OK (skipped=1)
2022-09-27T15:31:50.8435042Z 
2022-09-27T15:31:50.8435133Z Generating XML reports...
2022-09-27T15:31:50.8465205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153149.xml
2022-09-27T15:31:52.2011596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:52.2011974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:52.2013805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:52.2014166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:52.3354296Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:52.3363426Z 
2022-09-27T15:31:52.3363565Z Running tests...
2022-09-27T15:31:52.3364235Z ----------------------------------------------------------------------
2022-09-27T15:31:52.5740582Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2794
2022-09-27T15:31:52.5766238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2795
2022-09-27T15:31:52.5792217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2796
2022-09-27T15:31:52.5822918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2797
2022-09-27T15:31:53.7690868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:53.7691564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:53.7692429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:53.7693001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:53.8018664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:53.8019394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:53.8021186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:53.8021903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:53.8036860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:53.8037462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:53.8040575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:53.8041272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:53.8350295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:53.8351006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:53.8352364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:53.8353011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:53.9039430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:53.9373531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:31:53.9395355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:31:53.9674955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:54.3870013Z ok (2.050s)
2022-09-27T15:31:54.3870346Z 
2022-09-27T15:31:54.3870781Z ----------------------------------------------------------------------
2022-09-27T15:31:54.3871069Z Ran 1 test in 2.051s
2022-09-27T15:31:54.3871190Z 
2022-09-27T15:31:54.3871274Z OK
2022-09-27T15:31:54.3871381Z 
2022-09-27T15:31:54.3871499Z Generating XML reports...
2022-09-27T15:31:54.3904761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153152.xml
2022-09-27T15:31:55.7339218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:55.7339802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:55.7341301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:55.7341894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:55.8686718Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:55.8696660Z 
2022-09-27T15:31:55.8697395Z Running tests...
2022-09-27T15:31:55.8697832Z ----------------------------------------------------------------------
2022-09-27T15:31:56.1008333Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2861
2022-09-27T15:31:56.1034008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2862
2022-09-27T15:31:56.1060115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2863
2022-09-27T15:31:56.1091166Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2864
2022-09-27T15:31:57.2866223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:57.2866817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:57.2868052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:57.2868643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:57.2923607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:57.2924205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:57.2926366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:57.2926980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:57.2941596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:57.2942191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:57.2945074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:57.2945765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:57.3438844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:57.3439519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:57.3441563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:57.3442136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:57.4212902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:31:57.4269185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:31:57.4300608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:31:57.4879206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:31:57.8135788Z skip: Need at least 2 CUDA devices (1.944s)
2022-09-27T15:31:57.8136140Z 
2022-09-27T15:31:57.8136473Z ----------------------------------------------------------------------
2022-09-27T15:31:57.8136753Z Ran 1 test in 1.944s
2022-09-27T15:31:57.8136887Z 
2022-09-27T15:31:57.8136980Z OK (skipped=1)
2022-09-27T15:31:57.8137109Z 
2022-09-27T15:31:57.8137213Z Generating XML reports...
2022-09-27T15:31:57.8169344Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153155.xml
2022-09-27T15:31:59.1563437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:31:59.1563865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:31:59.1565733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:31:59.1566352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:31:59.2910418Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:31:59.2919458Z 
2022-09-27T15:31:59.2919909Z Running tests...
2022-09-27T15:31:59.2920535Z ----------------------------------------------------------------------
2022-09-27T15:31:59.5255677Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2916
2022-09-27T15:31:59.5281258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2917
2022-09-27T15:31:59.5307388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2918
2022-09-27T15:31:59.5338532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2919
2022-09-27T15:32:00.7109324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:00.7109886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:00.7111531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:00.7112163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:00.7395351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:00.7395968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:00.7397323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:00.7397925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:00.7459946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:00.7460635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:00.7462984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:00.7463676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:00.7528463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:00.7529106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:00.7531678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:00.7532307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:00.8453828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:00.8736364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:00.8827563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:00.8850887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:01.3383812Z ok (2.046s)
2022-09-27T15:32:01.3384172Z 
2022-09-27T15:32:01.3384661Z ----------------------------------------------------------------------
2022-09-27T15:32:01.3384934Z Ran 1 test in 2.046s
2022-09-27T15:32:01.3385054Z 
2022-09-27T15:32:01.3385102Z OK
2022-09-27T15:32:01.3385196Z 
2022-09-27T15:32:01.3385362Z Generating XML reports...
2022-09-27T15:32:01.3417468Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153159.xml
2022-09-27T15:32:02.6952905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:02.6953315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:02.6955323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:02.6955753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:02.8299421Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:02.8309126Z 
2022-09-27T15:32:02.8309256Z Running tests...
2022-09-27T15:32:02.8309836Z ----------------------------------------------------------------------
2022-09-27T15:32:03.0632216Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2983
2022-09-27T15:32:03.0657621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2984
2022-09-27T15:32:03.0683996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2985
2022-09-27T15:32:03.0715000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2986
2022-09-27T15:32:04.2750328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:04.2751069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:04.2751757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:04.2752264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:04.2967385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:04.2967945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:04.2969506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:04.2970034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:04.2985431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:04.2986118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:04.2989032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:04.2989624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:04.3159989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:04.3160618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:04.3163789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:04.3164395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:04.4090857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:04.4306677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:04.4320710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:04.4523459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:04.4801404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:32:04.4902067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T15:32:04.4904345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:32:04.4905000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:32:04.4905925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:04.4906620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:04.5005127Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:04.5005781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:04.8760851Z ok (2.045s)
2022-09-27T15:32:04.8761126Z 
2022-09-27T15:32:04.8761664Z ----------------------------------------------------------------------
2022-09-27T15:32:04.8761937Z Ran 1 test in 2.045s
2022-09-27T15:32:04.8762055Z 
2022-09-27T15:32:04.8762116Z OK
2022-09-27T15:32:04.8762208Z 
2022-09-27T15:32:04.8762305Z Generating XML reports...
2022-09-27T15:32:04.8793881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153202.xml
2022-09-27T15:32:06.2419007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:06.2419561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:06.2420302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:06.2420675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:06.3767205Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:06.3777079Z 
2022-09-27T15:32:06.3777154Z Running tests...
2022-09-27T15:32:06.3778127Z ----------------------------------------------------------------------
2022-09-27T15:32:06.6139004Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3050
2022-09-27T15:32:06.6163474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3051
2022-09-27T15:32:06.6190332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3052
2022-09-27T15:32:06.6221587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3053
2022-09-27T15:32:07.7464354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:07.7465009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:07.7466622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:07.7467260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:07.8146712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:07.8147407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:07.8148712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:07.8149222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:07.8242493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:07.8243160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:07.8245731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:07.8246374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:07.8352146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:07.8352728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:07.8356536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:07.8357479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:07.8815701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:07.9496162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:07.9646536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:07.9855582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:08.3265681Z ok (1.949s)
2022-09-27T15:32:08.3265851Z 
2022-09-27T15:32:08.3266283Z ----------------------------------------------------------------------
2022-09-27T15:32:08.3266563Z Ran 1 test in 1.949s
2022-09-27T15:32:08.3266694Z 
2022-09-27T15:32:08.3266769Z OK
2022-09-27T15:32:08.3266864Z 
2022-09-27T15:32:08.3266956Z Generating XML reports...
2022-09-27T15:32:08.3298931Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153206.xml
2022-09-27T15:32:09.6503285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:09.6504037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:09.6506272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:09.6506838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:09.7850699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:09.7860282Z 
2022-09-27T15:32:09.7860791Z Running tests...
2022-09-27T15:32:09.7861215Z ----------------------------------------------------------------------
2022-09-27T15:32:10.0183789Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3117
2022-09-27T15:32:10.0209875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3118
2022-09-27T15:32:10.0236078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3119
2022-09-27T15:32:10.0266765Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3120
2022-09-27T15:32:11.1724992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:11.1725564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:11.1727561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:11.1728078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:11.2045687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:11.2046279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:11.2048244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:11.2048831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:11.2673724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:11.2674350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:11.2675585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:11.2676234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:11.2748109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:11.2748757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:11.2752637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:11.2753303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:11.3078801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:11.3395105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:11.4041292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:11.4073028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:11.8313232Z ok (2.045s)
2022-09-27T15:32:11.8313785Z 
2022-09-27T15:32:11.8314470Z ----------------------------------------------------------------------
2022-09-27T15:32:11.8314918Z Ran 1 test in 2.045s
2022-09-27T15:32:11.8315083Z 
2022-09-27T15:32:11.8315148Z OK
2022-09-27T15:32:11.8315243Z 
2022-09-27T15:32:11.8315346Z Generating XML reports...
2022-09-27T15:32:11.8347290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153209.xml
2022-09-27T15:32:13.1843299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:13.1843684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:13.1845489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:13.1846038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:13.3191407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:13.3201073Z 
2022-09-27T15:32:13.3201206Z Running tests...
2022-09-27T15:32:13.3201788Z ----------------------------------------------------------------------
2022-09-27T15:32:13.5510043Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3184
2022-09-27T15:32:13.5552585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3185
2022-09-27T15:32:13.5572148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3186
2022-09-27T15:32:13.5603581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3187
2022-09-27T15:32:14.7362656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:14.7364083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:14.7371000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:14.7373070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:14.7389803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:14.7390325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:14.7390981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:14.7391505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:14.7567572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:14.7568227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:14.7570818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:14.7571406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:14.8130319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:14.8131215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:14.8132826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:14.8133596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:14.8722781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:14.8726309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:14.8901738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:14.9493018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:16.7672135Z ok (3.447s)
2022-09-27T15:32:16.7672354Z 
2022-09-27T15:32:16.7672707Z ----------------------------------------------------------------------
2022-09-27T15:32:16.7672966Z Ran 1 test in 3.447s
2022-09-27T15:32:16.7673102Z 
2022-09-27T15:32:16.7673151Z OK
2022-09-27T15:32:16.7673244Z 
2022-09-27T15:32:16.7673336Z Generating XML reports...
2022-09-27T15:32:16.7706018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153213.xml
2022-09-27T15:32:18.1149593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:18.1150120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:18.1151023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:18.1151459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:18.2612422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:18.2621766Z 
2022-09-27T15:32:18.2621886Z Running tests...
2022-09-27T15:32:18.2622362Z ----------------------------------------------------------------------
2022-09-27T15:32:18.4945405Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3275
2022-09-27T15:32:18.4971077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3276
2022-09-27T15:32:18.4997517Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3277
2022-09-27T15:32:18.5029595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3278
2022-09-27T15:32:19.6982440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:19.6983047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:19.6985150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:19.6985816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:19.7012011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:19.7012624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:19.7016028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:19.7016658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:19.7078987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:19.7079584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:19.7082540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:19.7083116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:19.7731379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:19.7731981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:19.7735525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:19.7736203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:19.8334813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:19.8352144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:19.8417705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:19.9376447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:20.2074731Z skip: Need at least 2 CUDA devices (1.945s)
2022-09-27T15:32:20.2074946Z 
2022-09-27T15:32:20.2075283Z ----------------------------------------------------------------------
2022-09-27T15:32:20.2075540Z Ran 1 test in 1.945s
2022-09-27T15:32:20.2075666Z 
2022-09-27T15:32:20.2075778Z OK (skipped=1)
2022-09-27T15:32:20.2075891Z 
2022-09-27T15:32:20.2075977Z Generating XML reports...
2022-09-27T15:32:20.2107248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153218.xml
2022-09-27T15:32:21.5587442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:21.5587820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:21.5590303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:21.5590697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:21.6931011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:21.6939753Z 
2022-09-27T15:32:21.6939895Z Running tests...
2022-09-27T15:32:21.6940495Z ----------------------------------------------------------------------
2022-09-27T15:32:21.9263750Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3330
2022-09-27T15:32:21.9289099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3331
2022-09-27T15:32:21.9315094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3332
2022-09-27T15:32:21.9346421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3333
2022-09-27T15:32:23.1262882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:23.1263486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:23.1265468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:23.1266117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:23.1560804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:23.1561440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:23.1563811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:23.1564404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:23.1782308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:23.1782967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:23.1785648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:23.1786304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:23.1857430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:23.1860733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:23.1861528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:23.1862151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:23.2614624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:23.2943507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:23.3111456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:23.3189013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:23.7392718Z ok (2.045s)
2022-09-27T15:32:23.7392967Z 
2022-09-27T15:32:23.7393466Z ----------------------------------------------------------------------
2022-09-27T15:32:23.7393752Z Ran 1 test in 2.045s
2022-09-27T15:32:23.7393873Z 
2022-09-27T15:32:23.7393936Z OK
2022-09-27T15:32:23.7394028Z 
2022-09-27T15:32:23.7394109Z Generating XML reports...
2022-09-27T15:32:23.7425696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153221.xml
2022-09-27T15:32:25.0715267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:25.0715631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:25.0717512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:25.0718047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:25.2061943Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:25.2071258Z 
2022-09-27T15:32:25.2071364Z Running tests...
2022-09-27T15:32:25.2071929Z ----------------------------------------------------------------------
2022-09-27T15:32:25.4395030Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3397
2022-09-27T15:32:25.4420530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3398
2022-09-27T15:32:25.4446830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3399
2022-09-27T15:32:25.4477923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3400
2022-09-27T15:32:26.6491103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:26.6491624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:26.6493603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:26.6494147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:26.6654973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:26.6655534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:26.6657557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:26.6658116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:26.6813662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:26.6814275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:26.6817088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:26.6817766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:26.6858180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:26.6858815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:26.6861952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:26.6862590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:26.7840511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:26.7998634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:26.8154548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:26.8227153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:27.1522951Z skip: Need at least 2 CUDA devices (1.945s)
2022-09-27T15:32:27.1523199Z 
2022-09-27T15:32:27.1523726Z ----------------------------------------------------------------------
2022-09-27T15:32:27.1524170Z Ran 1 test in 1.945s
2022-09-27T15:32:27.1524285Z 
2022-09-27T15:32:27.1524364Z OK (skipped=1)
2022-09-27T15:32:27.1524474Z 
2022-09-27T15:32:27.1525004Z Generating XML reports...
2022-09-27T15:32:27.1557914Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153225.xml
2022-09-27T15:32:28.4943361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:28.4943798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:28.4945173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:28.4945843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:28.6291559Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:28.6299619Z 
2022-09-27T15:32:28.6299963Z Running tests...
2022-09-27T15:32:28.6300600Z ----------------------------------------------------------------------
2022-09-27T15:32:28.8616745Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3452
2022-09-27T15:32:28.8642695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3453
2022-09-27T15:32:28.8669140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3454
2022-09-27T15:32:28.8700321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3455
2022-09-27T15:32:30.0505708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:30.0506299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:30.0507632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:30.0508209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:30.1000494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:30.1001081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:30.1002566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:30.1003123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:30.1624949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:30.1625884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:30.1627027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:30.1627688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:30.1847248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:30.1968365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:30.1968957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:30.1971985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:30.1972676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:30.2357769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:30.3301989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:30.3389177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:30.6747015Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:32:30.6747327Z 
2022-09-27T15:32:30.6747802Z ----------------------------------------------------------------------
2022-09-27T15:32:30.6748053Z Ran 1 test in 2.045s
2022-09-27T15:32:30.6748155Z 
2022-09-27T15:32:30.6748227Z OK (skipped=1)
2022-09-27T15:32:30.6748335Z 
2022-09-27T15:32:30.6748421Z Generating XML reports...
2022-09-27T15:32:30.6779239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153228.xml
2022-09-27T15:32:32.0502509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:32.0502883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:32.0504365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:32.0504725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:32.1846079Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:32.1854796Z 
2022-09-27T15:32:32.1854934Z Running tests...
2022-09-27T15:32:32.1855489Z ----------------------------------------------------------------------
2022-09-27T15:32:32.4174880Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3507
2022-09-27T15:32:32.4201316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3508
2022-09-27T15:32:32.4227735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3509
2022-09-27T15:32:32.4258281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3510
2022-09-27T15:32:33.5837309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:33.5837913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:33.5839567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:33.5840149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:33.5980990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:33.5981652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:33.5984438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:33.5985084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:33.6329739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:33.6330646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:33.6331603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:33.6332137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:33.7023386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:33.7024206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:33.7025392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:33.7026029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:33.7229187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:33.7394139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:33.7675367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:33.8370585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:34.2305423Z ok (2.045s)
2022-09-27T15:32:34.2305691Z 
2022-09-27T15:32:34.2306131Z ----------------------------------------------------------------------
2022-09-27T15:32:34.2306399Z Ran 1 test in 2.045s
2022-09-27T15:32:34.2306502Z 
2022-09-27T15:32:34.2306562Z OK
2022-09-27T15:32:34.2306654Z 
2022-09-27T15:32:34.2306747Z Generating XML reports...
2022-09-27T15:32:34.2338482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153232.xml
2022-09-27T15:32:35.5542705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:35.5543162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:35.5544429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:35.5544785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:35.6905846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:35.6906369Z 
2022-09-27T15:32:35.6906655Z Running tests...
2022-09-27T15:32:35.6907293Z ----------------------------------------------------------------------
2022-09-27T15:32:35.9246338Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3574
2022-09-27T15:32:35.9271436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3575
2022-09-27T15:32:35.9297295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3576
2022-09-27T15:32:35.9328265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3577
2022-09-27T15:32:37.0740801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:37.0741447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:37.0743494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:37.0744254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:37.0803644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:37.0804290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:37.0807850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:37.0808670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:37.1196572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:37.1197197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:37.1197940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:37.1198518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:37.1199238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:37.1199817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:37.1202947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:37.1203603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:37.2088782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:37.2143474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:37.2523347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:37.2568962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:37.6372649Z ok (1.947s)
2022-09-27T15:32:37.6372866Z 
2022-09-27T15:32:37.6373356Z ----------------------------------------------------------------------
2022-09-27T15:32:37.6373824Z Ran 1 test in 1.947s
2022-09-27T15:32:37.6374027Z 
2022-09-27T15:32:37.6374134Z OK
2022-09-27T15:32:37.6374249Z 
2022-09-27T15:32:37.6374360Z Generating XML reports...
2022-09-27T15:32:37.6406832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153235.xml
2022-09-27T15:32:38.9930600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:38.9931007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:38.9932751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:38.9933113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:39.1274390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:39.1283983Z 
2022-09-27T15:32:39.1284097Z Running tests...
2022-09-27T15:32:39.1285002Z ----------------------------------------------------------------------
2022-09-27T15:32:39.3635030Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3641
2022-09-27T15:32:39.3660025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3642
2022-09-27T15:32:39.3686102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3643
2022-09-27T15:32:39.3717781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3644
2022-09-27T15:32:40.5392961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:40.5393509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:40.5395136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:40.5395636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:40.5504668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:40.5505467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:40.5507747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:40.5508545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:40.5632499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:40.5633064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:40.5635786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:40.5636410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:40.5694834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:40.5695508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:40.5699014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:40.5699611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:40.6742741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:40.6847276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:40.6959234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:40.7055194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:40.7357319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:32:40.7458303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:32:40.7560322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T15:32:40.7561032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:32:40.7562103Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:40.7563070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:40.7563931Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:40.7564755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:32:41.0761315Z ok (1.947s)
2022-09-27T15:32:41.0761555Z 
2022-09-27T15:32:41.0762090Z ----------------------------------------------------------------------
2022-09-27T15:32:41.0762483Z Ran 1 test in 1.948s
2022-09-27T15:32:41.0762602Z 
2022-09-27T15:32:41.0762650Z OK
2022-09-27T15:32:41.0762753Z 
2022-09-27T15:32:41.0762845Z Generating XML reports...
2022-09-27T15:32:41.0795830Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153239.xml
2022-09-27T15:32:42.4160937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:42.4161411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:42.4162522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:42.4162954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:42.5504321Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:42.5514512Z 
2022-09-27T15:32:42.5514815Z Running tests...
2022-09-27T15:32:42.5515336Z ----------------------------------------------------------------------
2022-09-27T15:32:42.7889495Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3708
2022-09-27T15:32:42.7915239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3709
2022-09-27T15:32:42.7941133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3710
2022-09-27T15:32:42.7972466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3711
2022-09-27T15:32:43.9019242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:43.9019906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:43.9020986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:43.9021521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:43.9483669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:43.9484322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:43.9485954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:43.9486559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:43.9554325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:43.9554966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:43.9557303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:43.9557905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:43.9674783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:43.9675413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:43.9678010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:43.9678620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:44.0367688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:44.0862902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:44.0901895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:44.1005923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:44.5016133Z ok (1.950s)
2022-09-27T15:32:44.5016372Z 
2022-09-27T15:32:44.5016871Z ----------------------------------------------------------------------
2022-09-27T15:32:44.5017277Z Ran 1 test in 1.950s
2022-09-27T15:32:44.5017399Z 
2022-09-27T15:32:44.5017461Z OK
2022-09-27T15:32:44.5017553Z 
2022-09-27T15:32:44.5017643Z Generating XML reports...
2022-09-27T15:32:44.5050494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153242.xml
2022-09-27T15:32:45.8269965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:45.8270547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:45.8272637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:45.8273485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:45.9618505Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:45.9628349Z 
2022-09-27T15:32:45.9628879Z Running tests...
2022-09-27T15:32:45.9629511Z ----------------------------------------------------------------------
2022-09-27T15:32:46.1977800Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3775
2022-09-27T15:32:46.2003954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3776
2022-09-27T15:32:46.2030006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3777
2022-09-27T15:32:46.2063655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3778
2022-09-27T15:32:47.3921297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:47.3922032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:47.3923378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:47.3924091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:47.3975759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:47.3976401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:47.3978888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:47.3979533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:47.4001634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:47.4005056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:47.4005893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:47.4006505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:47.4259871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:47.4260483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:47.4262763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:47.4263380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:47.5270631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:47.5320050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:47.5375309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:47.5577685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:47.9140976Z ok (1.951s)
2022-09-27T15:32:47.9141238Z 
2022-09-27T15:32:47.9141781Z ----------------------------------------------------------------------
2022-09-27T15:32:47.9142120Z Ran 1 test in 1.951s
2022-09-27T15:32:47.9142237Z 
2022-09-27T15:32:47.9142300Z OK
2022-09-27T15:32:47.9142393Z 
2022-09-27T15:32:47.9142486Z Generating XML reports...
2022-09-27T15:32:47.9173893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153245.xml
2022-09-27T15:32:49.2365723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:49.2366107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:49.2368170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:49.2368540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:49.3704774Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:49.3714299Z 
2022-09-27T15:32:49.3714455Z Running tests...
2022-09-27T15:32:49.3714923Z ----------------------------------------------------------------------
2022-09-27T15:32:49.6009324Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3842
2022-09-27T15:32:49.6034175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3843
2022-09-27T15:32:49.6060136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3844
2022-09-27T15:32:49.6091984Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3845
2022-09-27T15:32:50.7634248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:50.7634891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:50.7635800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:50.7636311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:50.7637231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:50.7637832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:50.7640727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:50.7641393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:50.8060621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:50.8061245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:50.8062577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:50.8063200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:50.8422908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:50.8423565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:50.8425714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:50.8426372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:50.8989473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:50.8990007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:50.9407187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:50.9802380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:51.3136433Z skip: Need at least 1 CUDA device (1.942s)
2022-09-27T15:32:51.3136644Z 
2022-09-27T15:32:51.3136947Z ----------------------------------------------------------------------
2022-09-27T15:32:51.3137200Z Ran 1 test in 1.942s
2022-09-27T15:32:51.3137318Z 
2022-09-27T15:32:51.3137390Z OK (skipped=1)
2022-09-27T15:32:51.3137487Z 
2022-09-27T15:32:51.3137573Z Generating XML reports...
2022-09-27T15:32:51.3169734Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153249.xml
2022-09-27T15:32:52.6555802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:52.6556592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:52.6557521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:52.6558085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:52.7901046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:52.7910870Z 
2022-09-27T15:32:52.7911017Z Running tests...
2022-09-27T15:32:52.7911774Z ----------------------------------------------------------------------
2022-09-27T15:32:53.0247038Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3897
2022-09-27T15:32:53.0273291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3898
2022-09-27T15:32:53.0299990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3899
2022-09-27T15:32:53.0332728Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3900
2022-09-27T15:32:54.2011154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:54.2011812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:54.2013456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:54.2014019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:54.2701427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:54.2701964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:54.2704136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:54.2704688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:54.2767580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:54.2768226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:54.2772292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:54.2772973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:54.2857354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:54.2857989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:54.2861553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:54.2862212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:54.3360517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:54.4059379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:54.4173054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:54.4293772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:55.6391463Z ok (2.848s)
2022-09-27T15:32:55.6391710Z 
2022-09-27T15:32:55.6392274Z ----------------------------------------------------------------------
2022-09-27T15:32:55.6392714Z Ran 1 test in 2.848s
2022-09-27T15:32:55.6392944Z 
2022-09-27T15:32:55.6393060Z OK
2022-09-27T15:32:55.6393236Z 
2022-09-27T15:32:55.6393408Z Generating XML reports...
2022-09-27T15:32:55.6425289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153252.xml
2022-09-27T15:32:56.9822278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:56.9822877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:56.9824112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:56.9824680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:57.1170169Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:32:57.1179455Z 
2022-09-27T15:32:57.1179590Z Running tests...
2022-09-27T15:32:57.1180186Z ----------------------------------------------------------------------
2022-09-27T15:32:57.3505183Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3988
2022-09-27T15:32:57.3531817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3989
2022-09-27T15:32:57.3558351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3990
2022-09-27T15:32:57.3589775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3991
2022-09-27T15:32:58.5356886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:58.5357646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:58.5360204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:58.5377678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:58.5378728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:58.5379489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:58.5382620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:58.5383530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:58.5662581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:58.5662959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:58.5666893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:58.5667542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:58.6033538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:32:58.6034142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:32:58.6036573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:32:58.6037218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:32:58.6725667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:32:58.6820089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:32:58.7002074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:32:58.7392301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:32:59.7645365Z ok (2.646s)
2022-09-27T15:32:59.7645609Z 
2022-09-27T15:32:59.7646115Z ----------------------------------------------------------------------
2022-09-27T15:32:59.7646561Z Ran 1 test in 2.646s
2022-09-27T15:32:59.7646768Z 
2022-09-27T15:32:59.7646883Z OK
2022-09-27T15:32:59.7647048Z 
2022-09-27T15:32:59.7647217Z Generating XML reports...
2022-09-27T15:32:59.7679169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153257.xml
2022-09-27T15:33:01.1336058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:01.1336742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:01.1337697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:01.1338057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:01.2690736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:01.2700607Z 
2022-09-27T15:33:01.2700700Z Running tests...
2022-09-27T15:33:01.2701663Z ----------------------------------------------------------------------
2022-09-27T15:33:01.5016973Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4079
2022-09-27T15:33:01.5043379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4080
2022-09-27T15:33:01.5069049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4081
2022-09-27T15:33:01.5100574Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4082
2022-09-27T15:33:02.6784405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:02.6785061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:02.6786912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:02.6787477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:02.7002383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:02.7002991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:02.7005439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:02.7006074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:02.7437845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:02.7438533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:02.7440105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:02.7440776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:02.7776320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:02.7776937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:02.7778070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:02.7778637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:02.8157881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:02.8359900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:02.8785086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:02.9174106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:03.2147053Z skip: Need at least 2 CUDA devices (1.944s)
2022-09-27T15:33:03.2147439Z 
2022-09-27T15:33:03.2147786Z ----------------------------------------------------------------------
2022-09-27T15:33:03.2148057Z Ran 1 test in 1.944s
2022-09-27T15:33:03.2148172Z 
2022-09-27T15:33:03.2148482Z OK (skipped=1)
2022-09-27T15:33:03.2148581Z 
2022-09-27T15:33:03.2148671Z Generating XML reports...
2022-09-27T15:33:03.2181669Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153301.xml
2022-09-27T15:33:04.5603856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:04.5604452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:04.5605870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:04.5606458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:04.6950984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:04.6960559Z 
2022-09-27T15:33:04.6961101Z Running tests...
2022-09-27T15:33:04.6961540Z ----------------------------------------------------------------------
2022-09-27T15:33:04.9295282Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4134
2022-09-27T15:33:04.9320867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4135
2022-09-27T15:33:04.9347332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4136
2022-09-27T15:33:04.9379153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4137
2022-09-27T15:33:06.1233113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:06.1233760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:06.1234558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:06.1235163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:06.1235985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:06.1236540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:06.1238831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:06.1239404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:06.1255764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:06.1256399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:06.1259603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:06.1260213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:06.1269777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:06.1270452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:06.1274504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:06.1275171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:06.2573395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:06.2576875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:06.2586367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:06.2633700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:06.6424173Z ok (1.946s)
2022-09-27T15:33:06.6424380Z 
2022-09-27T15:33:06.6424967Z ----------------------------------------------------------------------
2022-09-27T15:33:06.6425281Z Ran 1 test in 1.946s
2022-09-27T15:33:06.6425417Z 
2022-09-27T15:33:06.6425480Z OK
2022-09-27T15:33:06.6425648Z 
2022-09-27T15:33:06.6425745Z Generating XML reports...
2022-09-27T15:33:06.6457334Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153304.xml
2022-09-27T15:33:07.9702792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:07.9703362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:07.9705855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:07.9706429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:08.1053391Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:08.1063040Z 
2022-09-27T15:33:08.1063536Z Running tests...
2022-09-27T15:33:08.1064138Z ----------------------------------------------------------------------
2022-09-27T15:33:08.3414541Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4201
2022-09-27T15:33:08.3439736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4202
2022-09-27T15:33:08.3467136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4203
2022-09-27T15:33:08.3497974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4204
2022-09-27T15:33:09.5680181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:09.5680781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:09.5682209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:09.5682810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:09.6126650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:09.6127279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:09.6129236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:09.6129795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:09.6380266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:09.6380911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:09.6381999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:09.6382629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:09.6715900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:09.6716567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:09.6717826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:09.6718457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:09.7030969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:09.7468633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:09.7708118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:09.8071856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:10.2545730Z ok (2.148s)
2022-09-27T15:33:10.2545947Z 
2022-09-27T15:33:10.2546284Z ----------------------------------------------------------------------
2022-09-27T15:33:10.2546632Z Ran 1 test in 2.148s
2022-09-27T15:33:10.2546763Z 
2022-09-27T15:33:10.2546824Z OK
2022-09-27T15:33:10.2546919Z 
2022-09-27T15:33:10.2547000Z Generating XML reports...
2022-09-27T15:33:10.2578725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153308.xml
2022-09-27T15:33:11.6067276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:11.6067664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:11.6069810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:11.6070168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:11.7407534Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:11.7417543Z 
2022-09-27T15:33:11.7418004Z Running tests...
2022-09-27T15:33:11.7418680Z ----------------------------------------------------------------------
2022-09-27T15:33:11.9730872Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4268
2022-09-27T15:33:11.9756848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4269
2022-09-27T15:33:11.9783738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4270
2022-09-27T15:33:11.9866171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4271
2022-09-27T15:33:13.1217446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:13.1217973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:13.1219942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:13.1220472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:13.1468102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:13.1468730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:13.1470074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:13.1470640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:13.1471723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:13.1472310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:13.1477091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:13.1477722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:13.1544033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:13.1544629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:13.1547311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:13.1547945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:13.2559013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:13.2799517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:13.2805563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:13.2914282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:13.5857883Z skip: Need at least 2 CUDA devices (1.844s)
2022-09-27T15:33:13.5858563Z 
2022-09-27T15:33:13.5858899Z ----------------------------------------------------------------------
2022-09-27T15:33:13.5859153Z Ran 1 test in 1.844s
2022-09-27T15:33:13.5859257Z 
2022-09-27T15:33:13.5859329Z OK (skipped=1)
2022-09-27T15:33:13.5859438Z 
2022-09-27T15:33:13.5859525Z Generating XML reports...
2022-09-27T15:33:13.5891248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153311.xml
2022-09-27T15:33:14.9237073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:14.9237666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:14.9239866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:14.9240441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:15.0588299Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:15.0598265Z 
2022-09-27T15:33:15.0598716Z Running tests...
2022-09-27T15:33:15.0599342Z ----------------------------------------------------------------------
2022-09-27T15:33:15.2975526Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4323
2022-09-27T15:33:15.3001493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4324
2022-09-27T15:33:15.3027848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4325
2022-09-27T15:33:15.3059595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4326
2022-09-27T15:33:16.4771329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:16.4771974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:16.4773530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:16.4774149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:16.4939963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:16.4940583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:16.4942271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:16.4942884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:16.5415344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:16.5415998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:16.5417202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:16.5417770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:16.5562738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:16.5563377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:16.5566385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:16.5567049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:16.6116313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:16.6325772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:16.6744028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:16.6878328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:17.1105875Z ok (2.050s)
2022-09-27T15:33:17.1106102Z 
2022-09-27T15:33:17.1106423Z ----------------------------------------------------------------------
2022-09-27T15:33:17.1106732Z Ran 1 test in 2.051s
2022-09-27T15:33:17.1106850Z 
2022-09-27T15:33:17.1106912Z OK
2022-09-27T15:33:17.1107005Z 
2022-09-27T15:33:17.1107100Z Generating XML reports...
2022-09-27T15:33:17.1139005Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153315.xml
2022-09-27T15:33:18.4797461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:18.4797852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:18.4800012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:18.4800416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:18.6156536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:18.6166099Z 
2022-09-27T15:33:18.6166194Z Running tests...
2022-09-27T15:33:18.6166919Z ----------------------------------------------------------------------
2022-09-27T15:33:18.8484305Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4390
2022-09-27T15:33:18.8509495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4391
2022-09-27T15:33:18.8535340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4392
2022-09-27T15:33:18.8566532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4393
2022-09-27T15:33:20.0281084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:20.0281792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:20.0285370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:20.0286037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:20.0701800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:20.0702406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:20.0704234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:20.0704813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:20.0755074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:20.0755705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:20.0758351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:20.0758984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:20.0835766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:20.0836367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:20.0839296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:20.0839930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:20.1622858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:20.2085428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:20.2087558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:20.2161690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:20.9617432Z ok (2.345s)
2022-09-27T15:33:20.9617664Z 
2022-09-27T15:33:20.9618146Z ----------------------------------------------------------------------
2022-09-27T15:33:20.9618550Z Ran 1 test in 2.345s
2022-09-27T15:33:20.9618669Z 
2022-09-27T15:33:20.9618731Z OK
2022-09-27T15:33:20.9618823Z 
2022-09-27T15:33:20.9618914Z Generating XML reports...
2022-09-27T15:33:20.9650532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153318.xml
2022-09-27T15:33:22.3272495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:22.3272943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:22.3274583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:22.3274952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:22.4615789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:22.4625162Z 
2022-09-27T15:33:22.4625352Z Running tests...
2022-09-27T15:33:22.4625695Z ----------------------------------------------------------------------
2022-09-27T15:33:22.6970949Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4481
2022-09-27T15:33:22.6997557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4482
2022-09-27T15:33:22.7023527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4483
2022-09-27T15:33:22.7054542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4484
2022-09-27T15:33:23.8509452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:23.8510336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:23.8511146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:23.8511768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:23.8512571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:23.8513210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:23.8514838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:23.8515435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:23.8844020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:23.8844650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:23.8846099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:23.8846711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:23.9043231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:23.9043823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:23.9045833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:23.9046728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:23.9864045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:23.9865079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:24.0186136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:24.0418891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:24.3097499Z skip: Need at least 2 CUDA devices (1.847s)
2022-09-27T15:33:24.3097830Z 
2022-09-27T15:33:24.3098360Z ----------------------------------------------------------------------
2022-09-27T15:33:24.3098824Z Ran 1 test in 1.847s
2022-09-27T15:33:24.3099032Z 
2022-09-27T15:33:24.3099168Z OK (skipped=1)
2022-09-27T15:33:24.3099293Z 
2022-09-27T15:33:24.3099382Z Generating XML reports...
2022-09-27T15:33:24.3131302Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153322.xml
2022-09-27T15:33:25.6796967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:25.6797507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:25.6798662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:25.6799058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:25.8144989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:25.8155093Z 
2022-09-27T15:33:25.8155220Z Running tests...
2022-09-27T15:33:25.8155822Z ----------------------------------------------------------------------
2022-09-27T15:33:26.0477641Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4536
2022-09-27T15:33:26.0504175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4537
2022-09-27T15:33:26.0530467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4538
2022-09-27T15:33:26.0562154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4539
2022-09-27T15:33:27.2195395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:27.2196096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:27.2197423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:27.2197952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:27.2213909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:27.2214514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:27.2217779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:27.2218453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:27.2238651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:27.2239272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:27.2242053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:27.2242688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:27.2502258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:27.2502922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:27.2507136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:27.2507801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:27.3532283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:27.3573630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:27.3608002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:27.3831573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:27.7661214Z ok (1.950s)
2022-09-27T15:33:27.7661461Z 
2022-09-27T15:33:27.7661978Z ----------------------------------------------------------------------
2022-09-27T15:33:27.7662301Z Ran 1 test in 1.951s
2022-09-27T15:33:27.7662418Z 
2022-09-27T15:33:27.7662480Z OK
2022-09-27T15:33:27.7662576Z 
2022-09-27T15:33:27.7662687Z Generating XML reports...
2022-09-27T15:33:27.7694989Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153325.xml
2022-09-27T15:33:29.0887771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:29.0888148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:29.0889974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:29.0890335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:29.2231182Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:29.2241093Z 
2022-09-27T15:33:29.2241403Z Running tests...
2022-09-27T15:33:29.2241731Z ----------------------------------------------------------------------
2022-09-27T15:33:29.4555360Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4603
2022-09-27T15:33:29.4580618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4604
2022-09-27T15:33:29.4606847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4605
2022-09-27T15:33:29.4637868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4606
2022-09-27T15:33:30.5979448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:30.5979993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:30.5981178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:30.5981653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:30.6273997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:30.6274616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:30.6275363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:30.6275946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:30.6395888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:30.6396564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:30.6399246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:30.6399887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:30.6730176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:30.6730760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:30.6732159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:30.6732930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:30.7340801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:30.7608224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:30.7729932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:30.8093855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:31.1682315Z ok (1.944s)
2022-09-27T15:33:31.1682559Z 
2022-09-27T15:33:31.1683035Z ----------------------------------------------------------------------
2022-09-27T15:33:31.1683302Z Ran 1 test in 1.944s
2022-09-27T15:33:31.1683430Z 
2022-09-27T15:33:31.1683492Z OK
2022-09-27T15:33:31.1683584Z 
2022-09-27T15:33:31.1683664Z Generating XML reports...
2022-09-27T15:33:31.1718152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153329.xml
2022-09-27T15:33:32.5029413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:32.5029771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:32.5031904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:32.5032269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:32.6380509Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:32.6391231Z 
2022-09-27T15:33:32.6391319Z Running tests...
2022-09-27T15:33:32.6392007Z ----------------------------------------------------------------------
2022-09-27T15:33:32.8736400Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4670
2022-09-27T15:33:32.8762485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4671
2022-09-27T15:33:32.8789352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4672
2022-09-27T15:33:32.8820894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4673
2022-09-27T15:33:34.0892888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:34.0893438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:34.0894226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:34.0894752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:34.1088508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:34.1089088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:34.1091891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:34.1092461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:34.1216137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:34.1216744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:34.1219631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:34.1220272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:34.1231553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:34.1232196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:34.1236125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:34.1236756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:34.2239279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:34.2473257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:34.2577135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:34.2599498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:34.5866683Z skip: Need at least 2 CUDA devices (1.947s)
2022-09-27T15:33:34.5866991Z 
2022-09-27T15:33:34.5867526Z ----------------------------------------------------------------------
2022-09-27T15:33:34.5867957Z Ran 1 test in 1.947s
2022-09-27T15:33:34.5868172Z 
2022-09-27T15:33:34.5868304Z OK (skipped=1)
2022-09-27T15:33:34.5868497Z 
2022-09-27T15:33:34.5868658Z Generating XML reports...
2022-09-27T15:33:34.5901837Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153332.xml
2022-09-27T15:33:35.9445595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:35.9445976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:35.9447793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:35.9448143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:36.0795354Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:36.0804302Z 
2022-09-27T15:33:36.0804401Z Running tests...
2022-09-27T15:33:36.0804884Z ----------------------------------------------------------------------
2022-09-27T15:33:36.3137892Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4725
2022-09-27T15:33:36.3163596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4726
2022-09-27T15:33:36.3190116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4727
2022-09-27T15:33:36.3222408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4728
2022-09-27T15:33:37.4965387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:37.4966013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:37.4967613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:37.4968216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:37.4990854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:37.4991503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:37.4994162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:37.4994827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:37.5116606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:37.5117223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:37.5120841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:37.5121486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:37.5361990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:37.5362876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:37.5364021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:37.5364648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:37.6307818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:37.6329334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:37.6487602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:37.6690407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:38.1268137Z ok (2.046s)
2022-09-27T15:33:38.1268389Z 
2022-09-27T15:33:38.1268859Z ----------------------------------------------------------------------
2022-09-27T15:33:38.1269248Z Ran 1 test in 2.046s
2022-09-27T15:33:38.1269414Z 
2022-09-27T15:33:38.1269503Z OK
2022-09-27T15:33:38.1269643Z 
2022-09-27T15:33:38.1269777Z Generating XML reports...
2022-09-27T15:33:38.1302368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153336.xml
2022-09-27T15:33:39.4727589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:39.4727960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:39.4730217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:39.4730582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:39.6074659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:39.6084029Z 
2022-09-27T15:33:39.6084344Z Running tests...
2022-09-27T15:33:39.6085070Z ----------------------------------------------------------------------
2022-09-27T15:33:39.8421828Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4792
2022-09-27T15:33:39.8446218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4793
2022-09-27T15:33:39.8473381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4794
2022-09-27T15:33:39.8505260Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4795
2022-09-27T15:33:41.0105511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:41.0106127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:41.0107280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:41.0107854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:41.0205820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:41.0206444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:41.0207950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:41.0208555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:41.0652936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:41.0653861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:41.0654822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:41.0655509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:41.0765067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:41.0765668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:41.0768246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:41.0768861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:41.1459563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:41.1559967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:41.1994217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:41.2134031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:41.6551714Z ok (2.046s)
2022-09-27T15:33:41.6551932Z 
2022-09-27T15:33:41.6553036Z ----------------------------------------------------------------------
2022-09-27T15:33:41.6553459Z Ran 1 test in 2.047s
2022-09-27T15:33:41.6553581Z 
2022-09-27T15:33:41.6553649Z OK
2022-09-27T15:33:41.6553735Z 
2022-09-27T15:33:41.6553831Z Generating XML reports...
2022-09-27T15:33:41.6585520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153339.xml
2022-09-27T15:33:43.0240334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:43.0240711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:43.0242016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:43.0242540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:43.1589071Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:43.1598322Z 
2022-09-27T15:33:43.1598456Z Running tests...
2022-09-27T15:33:43.1599106Z ----------------------------------------------------------------------
2022-09-27T15:33:43.3914348Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4859
2022-09-27T15:33:43.3938531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4860
2022-09-27T15:33:43.3965298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4861
2022-09-27T15:33:43.3998078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4862
2022-09-27T15:33:44.5612293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:44.5612889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:44.5614058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:44.5614569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:44.5648395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:44.5649033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:44.5651662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:44.5652203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:44.5777358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:44.5777882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:44.5780461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:44.5781115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:44.5926293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:44.5926921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:44.5929627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:44.5930241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:44.6949278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:44.6999540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:44.7156701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:44.7279475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:46.9072245Z ok (3.747s)
2022-09-27T15:33:46.9072512Z 
2022-09-27T15:33:46.9073017Z ----------------------------------------------------------------------
2022-09-27T15:33:46.9073269Z Ran 1 test in 3.747s
2022-09-27T15:33:46.9073373Z 
2022-09-27T15:33:46.9073434Z OK
2022-09-27T15:33:46.9073526Z 
2022-09-27T15:33:46.9073617Z Generating XML reports...
2022-09-27T15:33:46.9104765Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153343.xml
2022-09-27T15:33:48.2708937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:48.2709322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:48.2710462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:48.2710902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:48.4054912Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:48.4063533Z 
2022-09-27T15:33:48.4063668Z Running tests...
2022-09-27T15:33:48.4064391Z ----------------------------------------------------------------------
2022-09-27T15:33:48.6413744Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4950
2022-09-27T15:33:48.6439707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4951
2022-09-27T15:33:48.6465926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4952
2022-09-27T15:33:48.6497082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4953
2022-09-27T15:33:49.7603002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:49.7603648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:49.7605259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:49.7605831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:49.7837239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:49.7837868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:49.7839088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:49.7839979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:49.8434085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:49.8435033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:49.8436017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:49.8436579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:49.8475770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:49.8476442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:49.8479267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:49.8479843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:49.8952693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:49.9173735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:49.9787750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:49.9855493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:50.2611932Z skip: Need at least 2 CUDA devices (1.854s)
2022-09-27T15:33:50.2612196Z 
2022-09-27T15:33:50.2612637Z ----------------------------------------------------------------------
2022-09-27T15:33:50.2613053Z Ran 1 test in 1.855s
2022-09-27T15:33:50.2613232Z 
2022-09-27T15:33:50.2613321Z OK (skipped=1)
2022-09-27T15:33:50.2613489Z 
2022-09-27T15:33:50.2613616Z Generating XML reports...
2022-09-27T15:33:50.2646464Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153348.xml
2022-09-27T15:33:51.6292509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:51.6292907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:51.6294935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:51.6295301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:51.7651064Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:51.7660991Z 
2022-09-27T15:33:51.7661366Z Running tests...
2022-09-27T15:33:51.7661809Z ----------------------------------------------------------------------
2022-09-27T15:33:51.9990657Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5005
2022-09-27T15:33:52.0015894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5006
2022-09-27T15:33:52.0041874Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5007
2022-09-27T15:33:52.0073175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5008
2022-09-27T15:33:53.1561994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:53.1562674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:53.1564742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:53.1565325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:53.1849428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:53.1850162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:53.1851308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:53.1851931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:53.2779013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:53.2779667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:53.2782630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:53.2783356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:53.2919651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:53.2993574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:53.2993960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:53.2996050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:53.2996444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:53.3205289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:53.4185097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:53.4332515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:53.8118122Z ok (2.045s)
2022-09-27T15:33:53.8118310Z 
2022-09-27T15:33:53.8118692Z ----------------------------------------------------------------------
2022-09-27T15:33:53.8119003Z Ran 1 test in 2.046s
2022-09-27T15:33:53.8119129Z 
2022-09-27T15:33:53.8119192Z OK
2022-09-27T15:33:53.8119284Z 
2022-09-27T15:33:53.8119379Z Generating XML reports...
2022-09-27T15:33:53.8151265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153351.xml
2022-09-27T15:33:55.1414288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:55.1414694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:55.1416597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:55.1416967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:55.2760932Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:55.2770127Z 
2022-09-27T15:33:55.2770259Z Running tests...
2022-09-27T15:33:55.2771120Z ----------------------------------------------------------------------
2022-09-27T15:33:55.5103443Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5076
2022-09-27T15:33:55.5129835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5077
2022-09-27T15:33:55.5156346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5078
2022-09-27T15:33:55.5188780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5079
2022-09-27T15:33:56.6941017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:56.6941632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:56.6942390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:56.6942995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:56.6965395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:56.6966348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:56.6969460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:56.6970245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:56.7049872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:56.7050481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:56.7053686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:56.7054322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:56.7393647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:56.7394257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:56.7396858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:56.7397522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:56.8287319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:33:56.8306931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:33:56.8429037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:33:56.8716332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:33:57.2232638Z ok (1.946s)
2022-09-27T15:33:57.2232925Z 
2022-09-27T15:33:57.2233463Z ----------------------------------------------------------------------
2022-09-27T15:33:57.2233809Z Ran 1 test in 1.946s
2022-09-27T15:33:57.2233925Z 
2022-09-27T15:33:57.2233998Z OK
2022-09-27T15:33:57.2236013Z 
2022-09-27T15:33:57.2236414Z Generating XML reports...
2022-09-27T15:33:57.2266219Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153355.xml
2022-09-27T15:33:58.5526757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:33:58.5527211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:33:58.5528290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:33:58.5528714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:33:58.6871460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:33:58.6881168Z 
2022-09-27T15:33:58.6881266Z Running tests...
2022-09-27T15:33:58.6882283Z ----------------------------------------------------------------------
2022-09-27T15:33:58.9196755Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5143
2022-09-27T15:33:58.9222271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5144
2022-09-27T15:33:58.9248518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5145
2022-09-27T15:33:58.9280463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5146
2022-09-27T15:34:00.1188131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:00.1188720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:00.1190950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:00.1191548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:00.1253206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:00.1253822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:00.1256616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:00.1257190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:00.1356300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:00.1356914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:00.1359915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:00.1360547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:00.1489724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:00.1490329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:00.1492963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:00.1493578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:00.2537730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:00.2588988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:00.2691840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:00.2864356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:00.6325614Z skip: Need at least 2 CUDA devices (1.944s)
2022-09-27T15:34:00.6326033Z 
2022-09-27T15:34:00.6326382Z ----------------------------------------------------------------------
2022-09-27T15:34:00.6326637Z Ran 1 test in 1.944s
2022-09-27T15:34:00.6326738Z 
2022-09-27T15:34:00.6326815Z OK (skipped=1)
2022-09-27T15:34:00.6326932Z 
2022-09-27T15:34:00.6327018Z Generating XML reports...
2022-09-27T15:34:00.6359588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153358.xml
2022-09-27T15:34:01.9807203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:01.9807629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:01.9809412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:01.9809956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:02.1162338Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:02.1170638Z 
2022-09-27T15:34:02.1170760Z Running tests...
2022-09-27T15:34:02.1171336Z ----------------------------------------------------------------------
2022-09-27T15:34:02.3548070Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5198
2022-09-27T15:34:02.3574122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5199
2022-09-27T15:34:02.3600558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5200
2022-09-27T15:34:02.3631352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5201
2022-09-27T15:34:03.5410105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:03.5410662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:03.5412664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:03.5413164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:03.5647128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:03.5648026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:03.5650553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:03.5651258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:03.5684328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:03.5684944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:03.5688388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:03.5688968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:03.6105373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:03.6106009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:03.6109118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:03.6109763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:03.6794344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:03.7024903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:03.7039079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:03.7516311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:04.0676895Z ok (1.950s)
2022-09-27T15:34:04.0677113Z 
2022-09-27T15:34:04.0677457Z ----------------------------------------------------------------------
2022-09-27T15:34:04.0677719Z Ran 1 test in 1.951s
2022-09-27T15:34:04.0677837Z 
2022-09-27T15:34:04.0677899Z OK
2022-09-27T15:34:04.0677993Z 
2022-09-27T15:34:04.0678085Z Generating XML reports...
2022-09-27T15:34:04.0710097Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153402.xml
2022-09-27T15:34:05.4344141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:05.4344592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:05.4346269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:05.4346840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:05.5689064Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:05.5698336Z 
2022-09-27T15:34:05.5698464Z Running tests...
2022-09-27T15:34:05.5698898Z ----------------------------------------------------------------------
2022-09-27T15:34:05.8110589Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5265
2022-09-27T15:34:05.8136231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5266
2022-09-27T15:34:05.8163216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5267
2022-09-27T15:34:05.8195095Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5268
2022-09-27T15:34:06.9549487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:06.9550371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:06.9551869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:06.9552544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:06.9873878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:06.9874501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:06.9875300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:06.9875897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:06.9976290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:06.9976904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:06.9979381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:06.9980028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:07.0064554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:07.0065273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:07.0068255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:07.0068963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:07.0892952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:07.1237803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:07.1308213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:07.1434191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:09.0263096Z ok (3.456s)
2022-09-27T15:34:09.0263397Z 
2022-09-27T15:34:09.0264054Z ----------------------------------------------------------------------
2022-09-27T15:34:09.0264311Z Ran 1 test in 3.456s
2022-09-27T15:34:09.0264431Z 
2022-09-27T15:34:09.0264479Z OK
2022-09-27T15:34:09.0264571Z 
2022-09-27T15:34:09.0264664Z Generating XML reports...
2022-09-27T15:34:09.0296350Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153405.xml
2022-09-27T15:34:10.3708187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:10.3708592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:10.3709279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:10.3709637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:10.5049685Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:10.5059579Z 
2022-09-27T15:34:10.5059698Z Running tests...
2022-09-27T15:34:10.5060159Z ----------------------------------------------------------------------
2022-09-27T15:34:10.7391696Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5356
2022-09-27T15:34:10.7417705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5357
2022-09-27T15:34:10.7443326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5358
2022-09-27T15:34:10.7474905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5359
2022-09-27T15:34:11.8718422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:11.8719066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:11.8719890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:11.8720614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:11.9036233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:11.9036789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:11.9038618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:11.9039118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:11.9266465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:11.9267106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:11.9269018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:11.9269590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:11.9294099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:11.9294717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:11.9297516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:11.9298136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:12.0087546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:12.0404765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:12.0632432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:12.0672474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:12.3519360Z skip: Need at least 2 CUDA devices (1.846s)
2022-09-27T15:34:12.3519567Z 
2022-09-27T15:34:12.3519875Z ----------------------------------------------------------------------
2022-09-27T15:34:12.3520115Z Ran 1 test in 1.846s
2022-09-27T15:34:12.3520228Z 
2022-09-27T15:34:12.3520302Z OK (skipped=1)
2022-09-27T15:34:12.3520412Z 
2022-09-27T15:34:12.3520498Z Generating XML reports...
2022-09-27T15:34:12.3555263Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153410.xml
2022-09-27T15:34:13.7048337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:13.7072119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:13.7073065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:13.7073528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:13.8398854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:13.8407435Z 
2022-09-27T15:34:13.8407551Z Running tests...
2022-09-27T15:34:13.8408013Z ----------------------------------------------------------------------
2022-09-27T15:34:14.0822801Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5411
2022-09-27T15:34:14.0848547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5412
2022-09-27T15:34:14.0875663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5413
2022-09-27T15:34:14.0907974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5414
2022-09-27T15:34:15.2422959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:15.2424076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:15.2425762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:15.2426413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:15.2936184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:15.2936887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:15.2938343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:15.2939028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:15.2980626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:15.2981275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:15.2984548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:15.2985113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:15.3360040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:15.3360680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:15.3362813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:15.3363438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:15.3772877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:15.4317478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:15.4322084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:15.4680238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:15.8955166Z ok (2.054s)
2022-09-27T15:34:15.8955329Z 
2022-09-27T15:34:15.8955644Z ----------------------------------------------------------------------
2022-09-27T15:34:15.8955903Z Ran 1 test in 2.055s
2022-09-27T15:34:15.8956019Z 
2022-09-27T15:34:15.8956081Z OK
2022-09-27T15:34:15.8956173Z 
2022-09-27T15:34:15.8956266Z Generating XML reports...
2022-09-27T15:34:15.8988715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153413.xml
2022-09-27T15:34:17.2169175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:17.2169559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:17.2171478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:17.2171843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:17.3512406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:17.3521348Z 
2022-09-27T15:34:17.3521788Z Running tests...
2022-09-27T15:34:17.3522195Z ----------------------------------------------------------------------
2022-09-27T15:34:17.5841149Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5490
2022-09-27T15:34:17.5866288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5491
2022-09-27T15:34:17.5892518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5492
2022-09-27T15:34:17.5924391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5493
2022-09-27T15:34:18.7406712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:18.7407369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:18.7408365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:18.7408972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:18.7450685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:18.7451324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:18.7454134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:18.7454776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:18.7757445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:18.7758092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:18.7760619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:18.7761250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:18.8206854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:18.8207488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:18.8209130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:18.8209786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:18.8756132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:18.8792721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:18.9130628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:18.9526987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:19.5974367Z ok (2.245s)
2022-09-27T15:34:19.5974599Z 
2022-09-27T15:34:19.5975140Z ----------------------------------------------------------------------
2022-09-27T15:34:19.5975457Z Ran 1 test in 2.245s
2022-09-27T15:34:19.5975570Z 
2022-09-27T15:34:19.5975631Z OK
2022-09-27T15:34:19.5975723Z 
2022-09-27T15:34:19.5975815Z Generating XML reports...
2022-09-27T15:34:19.6011008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153417.xml
2022-09-27T15:34:20.9343047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:20.9343559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:20.9345045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:20.9345509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:21.0696402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:21.0706073Z 
2022-09-27T15:34:21.0706365Z Running tests...
2022-09-27T15:34:21.3104526Z ----------------------------------------------------------------------
2022-09-27T15:34:21.3105039Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5593
2022-09-27T15:34:21.3130389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5594
2022-09-27T15:34:21.3155939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5595
2022-09-27T15:34:21.3186812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5596
2022-09-27T15:34:22.4650673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:22.4651200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:22.4652600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:22.4653074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:22.4794411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:22.4795075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:22.4797737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:22.4798287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:22.4917895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:22.4918487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:22.4920412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:22.4920978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:22.4957084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:22.4957721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:22.4961245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:22.4972556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:22.5997798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:22.6141193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:22.6286758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:22.6291180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:23.0232072Z ok (1.952s)
2022-09-27T15:34:23.0232327Z 
2022-09-27T15:34:23.0232846Z ----------------------------------------------------------------------
2022-09-27T15:34:23.0233176Z Ran 1 test in 1.953s
2022-09-27T15:34:23.0233296Z 
2022-09-27T15:34:23.0233357Z OK
2022-09-27T15:34:23.0233449Z 
2022-09-27T15:34:23.0233558Z Generating XML reports...
2022-09-27T15:34:23.0268515Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153421.xml
2022-09-27T15:34:24.3593083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:24.3593747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:24.3594780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:24.3595388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:24.4932455Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:24.4942704Z 
2022-09-27T15:34:24.4942913Z Running tests...
2022-09-27T15:34:24.4943522Z ----------------------------------------------------------------------
2022-09-27T15:34:24.7269184Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5660
2022-09-27T15:34:24.7294773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5661
2022-09-27T15:34:24.7321445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5662
2022-09-27T15:34:24.7354021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5663
2022-09-27T15:34:25.9189346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:25.9190019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:25.9191217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:25.9191855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:25.9520845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:25.9521427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:25.9523534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:25.9524119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:25.9658473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:25.9659112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:25.9661930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:25.9662529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:25.9925109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:25.9925812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:25.9927903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:25.9928538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:26.0535113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:26.0864875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:26.1009426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:26.1306781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:26.4397808Z skip: Need at least 2 CUDA devices (1.945s)
2022-09-27T15:34:26.4399402Z 
2022-09-27T15:34:26.4400003Z ----------------------------------------------------------------------
2022-09-27T15:34:26.4400418Z Ran 1 test in 1.945s
2022-09-27T15:34:26.4400539Z 
2022-09-27T15:34:26.4400614Z OK (skipped=1)
2022-09-27T15:34:26.4400712Z 
2022-09-27T15:34:26.4400801Z Generating XML reports...
2022-09-27T15:34:26.4432644Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153424.xml
2022-09-27T15:34:27.7794192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:27.7794586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:27.7796238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:27.7796601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:27.9142922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:27.9152531Z 
2022-09-27T15:34:27.9152943Z Running tests...
2022-09-27T15:34:27.9153302Z ----------------------------------------------------------------------
2022-09-27T15:34:28.1508939Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5715
2022-09-27T15:34:28.1534567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5716
2022-09-27T15:34:28.1560860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5717
2022-09-27T15:34:28.1593661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5718
2022-09-27T15:34:29.3327423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:29.3328068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:29.3328817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:29.3329493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:29.3330249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:29.3330875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:29.3331670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:29.3332245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:29.4471555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:29.4472132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:29.4475172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:29.4475984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:29.4488906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:29.4490780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:29.4492228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:29.4492809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:29.4670731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:29.4850210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:29.5819592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:29.5850190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:30.0638488Z ok (2.148s)
2022-09-27T15:34:30.0638764Z 
2022-09-27T15:34:30.0639183Z ----------------------------------------------------------------------
2022-09-27T15:34:30.0639457Z Ran 1 test in 2.148s
2022-09-27T15:34:30.0639584Z 
2022-09-27T15:34:30.0639646Z OK
2022-09-27T15:34:30.0639739Z 
2022-09-27T15:34:30.0639821Z Generating XML reports...
2022-09-27T15:34:30.0674087Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153427.xml
2022-09-27T15:34:31.4117890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:31.4118505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:31.4119947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:31.4120508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:31.5476871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:31.5486448Z 
2022-09-27T15:34:31.5486787Z Running tests...
2022-09-27T15:34:31.5487442Z ----------------------------------------------------------------------
2022-09-27T15:34:31.7806853Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5782
2022-09-27T15:34:31.7833396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5783
2022-09-27T15:34:31.7859551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5784
2022-09-27T15:34:31.7891954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5785
2022-09-27T15:34:32.9737733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:32.9738262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:32.9740315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:32.9740769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:32.9902840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:32.9903556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:32.9905323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:32.9905998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:32.9990333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:32.9990981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:32.9994219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:32.9994876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:33.0395528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:33.0396187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:33.0397141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:33.0397738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:33.1091868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:33.1254952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:33.1348707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:33.1760893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:35.2964958Z ok (3.748s)
2022-09-27T15:34:35.2965249Z 
2022-09-27T15:34:35.2965762Z ----------------------------------------------------------------------
2022-09-27T15:34:35.2966122Z Ran 1 test in 3.748s
2022-09-27T15:34:35.2966243Z 
2022-09-27T15:34:35.2966306Z OK
2022-09-27T15:34:35.2966400Z 
2022-09-27T15:34:35.2966495Z Generating XML reports...
2022-09-27T15:34:35.2997990Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153431.xml
2022-09-27T15:34:36.6378602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:36.6379030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:36.6380457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:36.6381027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:36.7729649Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:36.7738782Z 
2022-09-27T15:34:36.7738884Z Running tests...
2022-09-27T15:34:36.7739317Z ----------------------------------------------------------------------
2022-09-27T15:34:36.7745743Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-09-27T15:34:36.7747180Z 
2022-09-27T15:34:36.7747523Z ----------------------------------------------------------------------
2022-09-27T15:34:36.7747852Z Ran 1 test in 0.001s
2022-09-27T15:34:36.7748068Z 
2022-09-27T15:34:36.7748216Z OK (skipped=1)
2022-09-27T15:34:36.7748403Z 
2022-09-27T15:34:36.7748570Z Generating XML reports...
2022-09-27T15:34:36.7770388Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153436.xml
2022-09-27T15:34:38.0026217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:38.0026663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:38.0028013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:38.0028541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:38.1378565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:38.1388291Z 
2022-09-27T15:34:38.1388416Z Running tests...
2022-09-27T15:34:38.1388973Z ----------------------------------------------------------------------
2022-09-27T15:34:38.3747257Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5883
2022-09-27T15:34:38.3772936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5884
2022-09-27T15:34:38.3799203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5885
2022-09-27T15:34:38.3829429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5886
2022-09-27T15:34:39.5574069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:39.5574778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:39.5575778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:39.5576437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:39.5847554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:39.5848293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:39.5849351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:39.5850043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:39.5938212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:39.5938810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:39.5941032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:39.5941609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:39.5981756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:39.5982411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:39.5986479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:39.5987120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:39.6920764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:39.7178774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:39.7272752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:39.7342999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:40.0873829Z ok (1.948s)
2022-09-27T15:34:40.0874095Z 
2022-09-27T15:34:40.0874405Z ----------------------------------------------------------------------
2022-09-27T15:34:40.0874663Z Ran 1 test in 1.948s
2022-09-27T15:34:40.0874781Z 
2022-09-27T15:34:40.0874844Z OK
2022-09-27T15:34:40.0874922Z 
2022-09-27T15:34:40.0875015Z Generating XML reports...
2022-09-27T15:34:40.0906828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153438.xml
2022-09-27T15:34:41.4169341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:41.4169754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:41.4171551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:41.4172122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:41.5518803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:41.5528041Z 
2022-09-27T15:34:41.5528339Z Running tests...
2022-09-27T15:34:41.5528955Z ----------------------------------------------------------------------
2022-09-27T15:34:41.5532717Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-09-27T15:34:41.5533093Z 
2022-09-27T15:34:41.5533421Z ----------------------------------------------------------------------
2022-09-27T15:34:41.5533671Z Ran 1 test in 0.000s
2022-09-27T15:34:41.5533788Z 
2022-09-27T15:34:41.5533862Z OK (skipped=1)
2022-09-27T15:34:41.5533987Z 
2022-09-27T15:34:41.5534114Z Generating XML reports...
2022-09-27T15:34:41.5557304Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153441.xml
2022-09-27T15:34:42.7783736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:42.7784341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:42.7785341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:42.7785892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:42.9133210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:42.9142455Z 
2022-09-27T15:34:42.9142577Z Running tests...
2022-09-27T15:34:42.9143221Z ----------------------------------------------------------------------
2022-09-27T15:34:43.1495070Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5960
2022-09-27T15:34:43.1521122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5961
2022-09-27T15:34:43.1547411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5962
2022-09-27T15:34:43.1579047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5963
2022-09-27T15:34:44.3209516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:44.3210145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:44.3211965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:44.3212521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:44.3422523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:44.3424709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:44.3425625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:44.3426233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:44.3439576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:44.3440217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:44.3442393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:44.3443037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:44.3459173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:44.3459781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:44.3462220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:44.3462853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:44.4550267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:44.4764920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:44.4793065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:44.4813487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:44.7622746Z skip: Need at least 2 CUDA devices (1.848s)
2022-09-27T15:34:44.7623064Z 
2022-09-27T15:34:44.7623574Z ----------------------------------------------------------------------
2022-09-27T15:34:44.7623997Z Ran 1 test in 1.848s
2022-09-27T15:34:44.7624169Z 
2022-09-27T15:34:44.7624249Z OK (skipped=1)
2022-09-27T15:34:44.7624364Z 
2022-09-27T15:34:44.7624437Z Generating XML reports...
2022-09-27T15:34:44.7656649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153442.xml
2022-09-27T15:34:46.1358587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:46.1358990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:46.1360521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:46.1360884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:46.2714284Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:46.2723454Z 
2022-09-27T15:34:46.2723537Z Running tests...
2022-09-27T15:34:46.2724583Z ----------------------------------------------------------------------
2022-09-27T15:34:46.5081349Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6015
2022-09-27T15:34:46.5107339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6016
2022-09-27T15:34:46.5133955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6017
2022-09-27T15:34:46.5166869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6018
2022-09-27T15:34:47.6973544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:47.6974159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:47.6975735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:47.6976417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:47.7431359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:47.7432072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:47.7432873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:47.7433499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:47.7434292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:47.7434943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:47.7435755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:47.7436409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:47.7789780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:47.7790351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:47.7793283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:47.7793952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:47.8320241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:34:47.8764020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:34:47.8808668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:34:47.9112521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:34:48.3211884Z ok (2.049s)
2022-09-27T15:34:48.3212151Z 
2022-09-27T15:34:48.3212582Z ----------------------------------------------------------------------
2022-09-27T15:34:48.3212951Z Ran 1 test in 2.049s
2022-09-27T15:34:48.3213137Z 
2022-09-27T15:34:48.3213239Z OK
2022-09-27T15:34:48.3213389Z 
2022-09-27T15:34:48.3213523Z Generating XML reports...
2022-09-27T15:34:48.3246619Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153446.xml
2022-09-27T15:34:49.6711063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:49.6711458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:49.6713432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:49.6713844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:49.8057068Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:49.8066883Z 
2022-09-27T15:34:49.8066975Z Running tests...
2022-09-27T15:34:49.8067483Z ----------------------------------------------------------------------
2022-09-27T15:34:49.8132402Z   test_forward_backward (__main__.ReducerTest) ... ok (0.006s)
2022-09-27T15:34:49.8204817Z 
2022-09-27T15:34:49.8205184Z ----------------------------------------------------------------------
2022-09-27T15:34:49.8205462Z Ran 1 test in 0.014s
2022-09-27T15:34:49.8205579Z 
2022-09-27T15:34:49.8205628Z OK
2022-09-27T15:34:49.8205722Z 
2022-09-27T15:34:49.8205816Z Generating XML reports...
2022-09-27T15:34:49.8228449Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153449.xml
2022-09-27T15:34:51.0510109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:51.0510744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:51.0512679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:51.0513027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:51.1886411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:51.1895818Z 
2022-09-27T15:34:51.1895943Z Running tests...
2022-09-27T15:34:51.1896423Z ----------------------------------------------------------------------
2022-09-27T15:34:51.1968939Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:34:51.1982485Z ok (0.009s)
2022-09-27T15:34:51.2031069Z 
2022-09-27T15:34:51.2031455Z ----------------------------------------------------------------------
2022-09-27T15:34:51.2031920Z Ran 1 test in 0.013s
2022-09-27T15:34:51.2032133Z 
2022-09-27T15:34:51.2032235Z OK
2022-09-27T15:34:51.2032350Z 
2022-09-27T15:34:51.2032442Z Generating XML reports...
2022-09-27T15:34:51.2055867Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153451.xml
2022-09-27T15:34:52.4709856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:52.4710455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:52.4712035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:52.4712617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:52.6059467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:52.6069606Z 
2022-09-27T15:34:52.6070110Z Running tests...
2022-09-27T15:34:52.6070539Z ----------------------------------------------------------------------
2022-09-27T15:34:52.6132727Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.006s)
2022-09-27T15:34:52.6200910Z 
2022-09-27T15:34:52.6201397Z ----------------------------------------------------------------------
2022-09-27T15:34:52.6201734Z Ran 1 test in 0.013s
2022-09-27T15:34:52.6201852Z 
2022-09-27T15:34:52.6201916Z OK
2022-09-27T15:34:52.6202009Z 
2022-09-27T15:34:52.6202099Z Generating XML reports...
2022-09-27T15:34:52.6225223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153452.xml
2022-09-27T15:34:53.8539745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:53.8540134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:53.8541853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:53.8542214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:53.9880390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:53.9889926Z 
2022-09-27T15:34:53.9890049Z Running tests...
2022-09-27T15:34:53.9890685Z ----------------------------------------------------------------------
2022-09-27T15:34:53.9928253Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-09-27T15:34:54.0017329Z 
2022-09-27T15:34:54.0017677Z ----------------------------------------------------------------------
2022-09-27T15:34:54.0018143Z Ran 1 test in 0.013s
2022-09-27T15:34:54.0018354Z 
2022-09-27T15:34:54.0018439Z OK
2022-09-27T15:34:54.0018540Z 
2022-09-27T15:34:54.0018630Z Generating XML reports...
2022-09-27T15:34:54.0042348Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153453.xml
2022-09-27T15:34:55.2368235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:55.2368614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:55.2370251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:55.2370614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:55.3715528Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:55.3724730Z 
2022-09-27T15:34:55.3724842Z Running tests...
2022-09-27T15:34:55.3725433Z ----------------------------------------------------------------------
2022-09-27T15:34:55.3791573Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-09-27T15:34:55.3856773Z 
2022-09-27T15:34:55.3857196Z ----------------------------------------------------------------------
2022-09-27T15:34:55.3857682Z Ran 1 test in 0.013s
2022-09-27T15:34:55.3857910Z 
2022-09-27T15:34:55.3858021Z OK
2022-09-27T15:34:55.3858108Z 
2022-09-27T15:34:55.3858202Z Generating XML reports...
2022-09-27T15:34:55.3881195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153455.xml
2022-09-27T15:34:56.6114705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:56.6115298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:56.6116694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:56.6117291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:56.7461936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:56.7471839Z 
2022-09-27T15:34:56.7472143Z Running tests...
2022-09-27T15:34:56.7472816Z ----------------------------------------------------------------------
2022-09-27T15:34:56.7514357Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-09-27T15:34:56.7604568Z 
2022-09-27T15:34:56.7604917Z ----------------------------------------------------------------------
2022-09-27T15:34:56.7605309Z Ran 1 test in 0.013s
2022-09-27T15:34:56.7605533Z 
2022-09-27T15:34:56.7605645Z OK
2022-09-27T15:34:56.7605814Z 
2022-09-27T15:34:56.7605972Z Generating XML reports...
2022-09-27T15:34:56.7628879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153456.xml
2022-09-27T15:34:57.9917506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:57.9917898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:57.9919880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:57.9920232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:58.1273880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:58.1283167Z 
2022-09-27T15:34:58.1283366Z Running tests...
2022-09-27T15:34:58.1283947Z ----------------------------------------------------------------------
2022-09-27T15:34:58.3624382Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:34:58.3625457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T15:34:58.3723198Z ok (0.244s)
2022-09-27T15:34:58.3723377Z 
2022-09-27T15:34:58.3723795Z ----------------------------------------------------------------------
2022-09-27T15:34:58.3724216Z Ran 1 test in 0.244s
2022-09-27T15:34:58.3724428Z 
2022-09-27T15:34:58.3724552Z OK
2022-09-27T15:34:58.3724708Z 
2022-09-27T15:34:58.3724804Z Generating XML reports...
2022-09-27T15:34:58.3748464Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220927153458.xml
2022-09-27T15:34:59.6606148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:34:59.6606651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:34:59.6607797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:34:59.6608343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:34:59.7946113Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:34:59.7955290Z 
2022-09-27T15:34:59.7955404Z Running tests...
2022-09-27T15:34:59.7956006Z ----------------------------------------------------------------------
2022-09-27T15:35:00.0192845Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.223s)
2022-09-27T15:35:00.0193338Z 
2022-09-27T15:35:00.0193555Z ----------------------------------------------------------------------
2022-09-27T15:35:00.0193809Z Ran 1 test in 0.224s
2022-09-27T15:35:00.0193933Z 
2022-09-27T15:35:00.0194008Z OK (skipped=1)
2022-09-27T15:35:00.0194119Z 
2022-09-27T15:35:00.0194203Z Generating XML reports...
2022-09-27T15:35:00.0218518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220927153459.xml
2022-09-27T15:35:00.4702861Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-09-27 15:35:00.469806]
2022-09-27T15:35:00.4703466Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:35:00.469888]
2022-09-27T15:35:01.8799812Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-09-27T15:35:01.8815840Z 
2022-09-27T15:35:01.8816299Z Running tests...
2022-09-27T15:35:01.8816704Z ----------------------------------------------------------------------
2022-09-27T15:35:01.8981349Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6194
2022-09-27T15:35:01.9007897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6195
2022-09-27T15:35:03.0209345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:03.0209712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:03.0213236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:03.0213782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:03.0319119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:03.0319781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:03.0323598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:03.0324119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:03.3706641Z dist init r=1, world=2
2022-09-27T15:35:03.3878700Z dist init r=0, world=2
2022-09-27T15:35:03.4087271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:03.4087735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:03.4088446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:03.4088971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:03.4093726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:03.4094388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:03.7046032Z skip: Need at least 2 CUDA devices (1.823s)
2022-09-27T15:35:03.7091526Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6222
2022-09-27T15:35:03.7120532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6223
2022-09-27T15:35:04.8266134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:04.8266509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:04.8269325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:04.8269861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:04.8402311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:04.8402684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:04.8406795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:04.8407246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:05.1743352Z dist init r=0, world=2
2022-09-27T15:35:05.1875531Z dist init r=1, world=2
2022-09-27T15:35:05.1983186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:05.1984138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:05.1984782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:05.1985362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:05.2088995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:05.2089606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:05.5155787Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:05.5201364Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6250
2022-09-27T15:35:05.5230422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6251
2022-09-27T15:35:06.5821573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:06.5821950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:06.5825184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:06.5826095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:06.6043456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:06.6044098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:06.6047812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:06.6048454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:06.9279377Z dist init r=0, world=2
2022-09-27T15:35:06.9559564Z dist init r=1, world=2
2022-09-27T15:35:06.9688640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:06.9689145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:06.9689873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:06.9690410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:06.9795068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:06.9795649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:07.3264323Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:07.3312672Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6278
2022-09-27T15:35:07.3341865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6279
2022-09-27T15:35:08.4334277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:08.4334963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:08.4338384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:08.4339018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:08.4362346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:08.4362928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:08.4367034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:08.4367680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:08.7828867Z dist init r=0, world=2
2022-09-27T15:35:08.7852867Z dist init r=1, world=2
2022-09-27T15:35:08.7961281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:08.7962004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:08.7962645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:08.7963177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:08.8067039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:08.8067639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:09.1377205Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:09.1423440Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6306
2022-09-27T15:35:09.1453292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6307
2022-09-27T15:35:10.2529069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:10.2529449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:10.2532178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:10.2532530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:10.2754096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:10.2754772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:10.2758449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:10.2759082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:10.6093820Z dist init r=0, world=2
2022-09-27T15:35:10.6219651Z dist init r=1, world=2
2022-09-27T15:35:10.6327732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:10.6328314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:10.6329015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:10.6329545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:10.6432736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:10.6433289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:10.9487023Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:10.9537177Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6334
2022-09-27T15:35:10.9566641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6335
2022-09-27T15:35:12.0587530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:12.0588209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:12.0591532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:12.0591937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:12.0662370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:12.0662858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:12.0666871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:12.0667418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:12.4130017Z dist init r=0, world=2
2022-09-27T15:35:12.4130752Z dist init r=1, world=2
2022-09-27T15:35:12.4239291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:12.4239764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:12.4240729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:12.4241267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:12.4345722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:12.4346272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:12.7600319Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:12.7646019Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6362
2022-09-27T15:35:12.7674956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6363
2022-09-27T15:35:13.8642753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:13.8643134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:13.8646009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:13.8646520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:13.8754148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:13.8754589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:13.8758452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:13.8758969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:14.2110087Z dist init r=0, world=2
2022-09-27T15:35:14.2289441Z dist init r=1, world=2
2022-09-27T15:35:14.2397869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:14.2398453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:14.2399108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:14.2399928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:14.2403249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:14.2403771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:14.5708981Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:14.5758916Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6390
2022-09-27T15:35:14.5787861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6391
2022-09-27T15:35:15.6397751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:15.6398137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:15.6401155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:15.6401535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:15.6612132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:15.6612512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:15.6616621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:15.6617453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:15.9859553Z dist init r=0, world=2
2022-09-27T15:35:16.0133665Z dist init r=1, world=2
2022-09-27T15:35:16.0242299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:16.0242936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:16.0243686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:16.0244539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:16.0247880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:16.0248469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:16.3821303Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:16.3867221Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6418
2022-09-27T15:35:16.3896350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6419
2022-09-27T15:35:17.4857652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:17.4858027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:17.4861375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:17.4861763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:17.4932759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:17.4933119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:17.4937072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:17.4937478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:17.8333683Z dist init r=0, world=2
2022-09-27T15:35:17.8394878Z dist init r=1, world=2
2022-09-27T15:35:17.8503410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:17.8504289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:17.8504909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:17.8505510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:17.8608593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:17.8609022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:18.1929485Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:18.1975912Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6446
2022-09-27T15:35:18.2004756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6447
2022-09-27T15:35:19.2946424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:19.2946846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:19.2949752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:19.2950375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:19.3072936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:19.3073441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:19.3076227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:19.3076586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:19.6527152Z dist init r=0, world=2
2022-09-27T15:35:19.6570960Z dist init r=1, world=2
2022-09-27T15:35:19.6736051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:19.6736498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:19.6737216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:19.6737739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:19.6841725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:19.6842179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:20.0038682Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:20.0086640Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6474
2022-09-27T15:35:20.0116696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6475
2022-09-27T15:35:21.0906224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:21.0906641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:21.0908985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:21.0909566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:21.1328151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:21.1328556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:21.1331835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:21.1332363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:21.4378779Z dist init r=1, world=2
2022-09-27T15:35:21.4780493Z dist init r=0, world=2
2022-09-27T15:35:21.4888161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:21.4890406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:21.4891104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:21.4991473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:21.4995206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:21.4995858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:21.8152324Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:21.8197212Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6502
2022-09-27T15:35:21.8226323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6503
2022-09-27T15:35:22.9090291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:22.9090945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:22.9092595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:22.9093168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:22.9214179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:22.9214676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:22.9218475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:22.9218996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:23.2576956Z dist init r=0, world=2
2022-09-27T15:35:23.2697751Z dist init r=1, world=2
2022-09-27T15:35:23.2805601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:23.2806346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:23.2806962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:23.2807484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:23.2911505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:23.2912154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:23.6260890Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:23.6309435Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6530
2022-09-27T15:35:23.6338561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6531
2022-09-27T15:35:24.7229532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:24.7229905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:24.7233342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:24.7233733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:24.7309074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:24.7309408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:24.7313456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:24.7313839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:25.0780264Z dist init r=1, world=2
2022-09-27T15:35:25.0813553Z dist init r=0, world=2
2022-09-27T15:35:25.1023053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:25.1023748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:25.1024682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:25.1025212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:25.1129025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:25.1129492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:25.4373419Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:25.4419778Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6558
2022-09-27T15:35:25.4448709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6559
2022-09-27T15:35:26.5432014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:26.5432594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:26.5434946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:26.5435519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:26.5550148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:26.5550764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:26.5554289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:26.5554897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:26.9013618Z dist init r=0, world=2
2022-09-27T15:35:26.9013961Z dist init r=1, world=2
2022-09-27T15:35:26.9120551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:26.9123120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:26.9124255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:26.9222602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:26.9227068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:26.9227746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:27.2482623Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:27.2529174Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6586
2022-09-27T15:35:27.2558304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6587
2022-09-27T15:35:28.3491529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:28.3492188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:28.3492977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:28.3493497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:28.3494663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:28.3495276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:28.3496031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:28.3496579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:28.6969463Z dist init r=1, world=2
2022-09-27T15:35:28.7019100Z dist init r=0, world=2
2022-09-27T15:35:28.7228962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:28.7229796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:28.7230414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:28.7231023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:28.7233363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:28.7234103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:29.0593342Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:29.0640176Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6614
2022-09-27T15:35:29.0669279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6615
2022-09-27T15:35:30.1327746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:30.1328147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:30.1331052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:30.1331418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:30.1940051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:30.1940407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:30.1943963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:30.1944342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:30.4772664Z dist init r=1, world=2
2022-09-27T15:35:30.5473133Z dist init r=0, world=2
2022-09-27T15:35:30.5585556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:30.5586895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:30.5587638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:30.5588169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:30.5591875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:30.5592426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:30.8707856Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:30.8753913Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6642
2022-09-27T15:35:30.8783446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6643
2022-09-27T15:35:31.9419141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:31.9419791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:31.9423060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:31.9423661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:31.9645698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:31.9646329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:31.9650100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:31.9650770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:32.2903615Z dist init r=0, world=2
2022-09-27T15:35:32.3181832Z dist init r=1, world=2
2022-09-27T15:35:32.3290340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:32.3290931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:32.3291610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:32.3292147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:32.3396136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:32.3396707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:32.6816331Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:32.6865725Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6670
2022-09-27T15:35:32.6894708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6671
2022-09-27T15:35:33.7825125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:33.7825582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:33.7829513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:33.7829892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:33.7990343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:33.7990710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:33.7994513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:33.7994853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:34.1285234Z dist init r=1, world=2
2022-09-27T15:35:34.1477118Z dist init r=0, world=2
2022-09-27T15:35:34.1686617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:34.1687040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:34.1687763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:34.1688307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:34.1691872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:34.1692273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:34.4927731Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:34.4975635Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6698
2022-09-27T15:35:34.5005173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6699
2022-09-27T15:35:35.5618447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:35.5619477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:35.5620701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:35.5621535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:35.5842217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:35.5842924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:35.5847107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:35.5847752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:35.9073039Z dist init r=1, world=2
2022-09-27T15:35:35.9364815Z dist init r=0, world=2
2022-09-27T15:35:35.9481946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:35.9482594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:35.9483222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:35.9483754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:35.9587939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:35.9588317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:36.3038256Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:36.3084891Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6726
2022-09-27T15:35:36.3114505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6727
2022-09-27T15:35:37.4176846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:37.4177235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:37.4180115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:37.4180480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:37.4235180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:37.4235525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:37.4239770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:37.4240125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:37.7643851Z dist init r=0, world=2
2022-09-27T15:35:37.7754590Z dist init r=1, world=2
2022-09-27T15:35:37.7952551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:37.7953280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:37.7954112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:37.7954710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:37.7958461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:37.7958933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:38.1148967Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:38.1194986Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6754
2022-09-27T15:35:38.1224703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6755
2022-09-27T15:35:39.2355327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:39.2355981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:39.2358486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:39.2358859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:39.2423745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:39.2424298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:39.2428633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:39.2429031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:39.5834218Z dist init r=0, world=2
2022-09-27T15:35:39.5882634Z dist init r=1, world=2
2022-09-27T15:35:39.6041874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:39.6042272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:39.6042906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:39.6043439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:39.6147931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:39.6148311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:39.9263654Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:39.9310950Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6782
2022-09-27T15:35:39.9339209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6783
2022-09-27T15:35:41.0442977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:41.0443396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:41.0446749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:41.0447222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:41.0491183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:41.0491725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:41.0495573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:41.0496117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:41.3983434Z dist init r=0, world=2
2022-09-27T15:35:41.3989582Z dist init r=1, world=2
2022-09-27T15:35:41.4192097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:41.4192815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:41.4193644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:41.4194182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:41.4198053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:41.4199226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:41.7372399Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:41.7418775Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6810
2022-09-27T15:35:41.7447658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6811
2022-09-27T15:35:42.8823316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:42.8823704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:42.8827145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:42.8827659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:42.8828364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:42.8828757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:42.8833243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:42.8833710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:43.2326354Z dist init r=0, world=2
2022-09-27T15:35:43.2326568Z dist init r=1, world=2
2022-09-27T15:35:43.2433889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:43.2436160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:43.2437437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:43.2535676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:43.2539754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:43.2540305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:43.5482127Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:43.5528263Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6838
2022-09-27T15:35:43.5558072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6839
2022-09-27T15:35:44.6536234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:44.6536646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:44.6539039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:44.6539637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:44.6894150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:44.6894522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:44.6897379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:44.6897934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:44.9995684Z dist init r=0, world=2
2022-09-27T15:35:45.0428337Z dist init r=1, world=2
2022-09-27T15:35:45.0536247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:45.0537125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:45.0537731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:45.0538269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:45.0642700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:45.0643271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:45.3592299Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:45.3637720Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6866
2022-09-27T15:35:45.3667743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6867
2022-09-27T15:35:46.4776370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:46.4776766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:46.4779440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:46.4779809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:46.4828422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:46.4828769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:46.4832668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:46.4833053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:46.8242439Z dist init r=0, world=2
2022-09-27T15:35:46.8259195Z dist init r=1, world=2
2022-09-27T15:35:46.8450125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:46.8450593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:46.8451236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:46.8451769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:46.8555774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:46.8556304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:47.1705660Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:47.1751685Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6894
2022-09-27T15:35:47.1780661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6895
2022-09-27T15:35:48.2520781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:48.2521164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:48.2523996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:48.2524603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:48.2638821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:48.2639405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:48.2642964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:48.2643467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:48.6035277Z dist init r=1, world=2
2022-09-27T15:35:48.6117489Z dist init r=0, world=2
2022-09-27T15:35:48.6326928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:48.6327357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:48.6328049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:48.6328588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:48.6433207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:48.6433835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:48.9818451Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:48.9865809Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6922
2022-09-27T15:35:48.9895048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6923
2022-09-27T15:35:50.1049376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:50.1049818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:50.1052126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:50.1052609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:50.1252863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:50.1253391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:50.1256911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:50.1257473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:50.4576316Z dist init r=0, world=2
2022-09-27T15:35:50.4747022Z dist init r=1, world=2
2022-09-27T15:35:50.4855127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:50.4855822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:50.4856617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:50.4857163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:50.4960459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:50.8932062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:50.8932421Z skip: Need at least 2 CUDA devices (1.911s)
2022-09-27T15:35:50.8979769Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6950
2022-09-27T15:35:50.9009146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6951
2022-09-27T15:35:51.9758823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:51.9759436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:51.9762590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:51.9762984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:52.0292728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:52.0293188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:52.0296297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:52.0296804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:52.3215663Z dist init r=0, world=2
2022-09-27T15:35:52.3757403Z dist init r=1, world=2
2022-09-27T15:35:52.3926917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:52.3927374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:52.3928001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:52.3928535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:52.4032618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:52.4033266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:52.7043787Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:52.7090502Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6978
2022-09-27T15:35:52.7119701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6979
2022-09-27T15:35:53.8142450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:53.8142910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:53.8144752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:53.8145398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:53.8361984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:53.8362535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:53.8366161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:53.8366690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:54.1607200Z dist init r=1, world=2
2022-09-27T15:35:54.1924755Z dist init r=0, world=2
2022-09-27T15:35:54.2133872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:54.2134284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:54.2134912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:54.2135451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:54.2140612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:54.5153472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:54.5154077Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:54.5201628Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7006
2022-09-27T15:35:54.5231621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7007
2022-09-27T15:35:55.6127280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:55.6127735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:55.6130535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:55.6131160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:55.6418657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:55.6419045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:55.6422855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:55.6423231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:55.9614279Z dist init r=1, world=2
2022-09-27T15:35:55.9892937Z dist init r=0, world=2
2022-09-27T15:35:56.0102568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:56.0103310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:56.0104147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:56.0104676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:56.0107434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:56.0107993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:56.3266352Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:56.3313769Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7034
2022-09-27T15:35:56.3342330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7035
2022-09-27T15:35:57.3925421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:57.3925827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:57.3929127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:57.3929526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:57.4136942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:57.4137343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:57.4141167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:57.4141694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:57.7481933Z dist init r=0, world=2
2022-09-27T15:35:57.7588415Z dist init r=1, world=2
2022-09-27T15:35:57.7789781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:57.7799054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:57.7799770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:57.7800386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:57.7895216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:57.7895872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:58.1378608Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:58.1425200Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7062
2022-09-27T15:35:58.1454271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7063
2022-09-27T15:35:59.2316420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:59.2317105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:59.2320570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:59.2321187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:59.2551106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:35:59.2551759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:35:59.2554149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:35:59.2554832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:35:59.5786231Z dist init r=1, world=2
2022-09-27T15:35:59.6091874Z dist init r=0, world=2
2022-09-27T15:35:59.6296931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:35:59.6297633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:35:59.6298248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:59.6298780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:35:59.6402466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:35:59.6403036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:35:59.9488384Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:35:59.9533863Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7090
2022-09-27T15:35:59.9562519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7091
2022-09-27T15:36:01.0574044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:01.0574412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:01.0578064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:01.0578431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:01.0617297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:01.0617894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:01.0621846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:01.0622427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:01.4103518Z dist init r=1, world=2
2022-09-27T15:36:01.4127340Z dist init r=0, world=2
2022-09-27T15:36:01.4312079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:01.4312753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:01.4313388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:01.4313925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:01.4417972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:01.4418533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:01.7596986Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:36:01.7647802Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7118
2022-09-27T15:36:01.7677246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7119
2022-09-27T15:36:02.8292575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:02.8293248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:02.8296448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:02.8297054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:02.8497227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:02.8497859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:02.8500826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:02.8501461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:03.1798674Z dist init r=0, world=2
2022-09-27T15:36:03.2042779Z dist init r=1, world=2
2022-09-27T15:36:03.2151039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:03.2151570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:03.2152199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:03.2152746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:03.2257095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:03.2257604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:03.5715553Z skip: Need at least 2 CUDA devices (1.812s)
2022-09-27T15:36:03.5729430Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-09-27T15:36:03.5730476Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.002s)
2022-09-27T15:36:03.5788721Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7146
2022-09-27T15:36:03.5818251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7147
2022-09-27T15:36:04.6878951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:04.6879342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:04.6881777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:04.6882240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:04.7016340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:04.7016680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:04.7021007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:04.7021411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:05.0392033Z dist init r=1, world=2
2022-09-27T15:36:05.0597600Z dist init r=0, world=2
2022-09-27T15:36:05.0806937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:05.0807679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:05.0808392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:05.0809237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:05.0812945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:05.0813518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:05.3851758Z skip: Need at least 2 CUDA devices (1.812s)
2022-09-27T15:36:05.3909036Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7174
2022-09-27T15:36:05.3938075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7175
2022-09-27T15:36:06.5104955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:06.5105722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:06.5109205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:06.5109812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:06.5326937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:06.5327574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:06.5330707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:06.5331342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:06.8565319Z dist init r=0, world=2
2022-09-27T15:36:06.8767266Z dist init r=1, world=2
2022-09-27T15:36:06.8874670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:06.8877150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:06.8877945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:06.8976853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:06.8981384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:06.8982028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:07.1973095Z skip: Need at least 2 CUDA devices (1.812s)
2022-09-27T15:36:07.2017529Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7202
2022-09-27T15:36:07.2048359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7203
2022-09-27T15:36:08.2735499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:08.2735895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:08.2737914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:08.2738429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:08.2864148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:08.2864528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:08.2867922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:08.2868306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:08.6189737Z dist init r=1, world=2
2022-09-27T15:36:08.6341714Z dist init r=0, world=2
2022-09-27T15:36:08.6498686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:08.6499145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:08.6499794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:08.6500355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:08.6504574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:08.6505210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:09.0082358Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:36:09.0146229Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7230
2022-09-27T15:36:09.0178595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7231
2022-09-27T15:36:10.0777771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:10.0778453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:10.0782034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:10.0782661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:10.1022888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:10.1023537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:10.1027683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:10.1028307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:10.4311329Z dist init r=1, world=2
2022-09-27T15:36:10.4550189Z dist init r=0, world=2
2022-09-27T15:36:10.4759384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:10.4760099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:10.4760772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:10.4761590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:10.4764977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:10.4765507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:10.8211883Z skip: Need at least 2 CUDA devices (1.813s)
2022-09-27T15:36:10.8257399Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7258
2022-09-27T15:36:10.8286551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7259
2022-09-27T15:36:11.9153248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:11.9153897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:11.9157473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:11.9158088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:11.9862053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:11.9862710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:11.9865874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:11.9866530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:12.2635840Z dist init r=0, world=2
2022-09-27T15:36:12.3405477Z dist init r=1, world=2
2022-09-27T15:36:12.3513643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:12.3514233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:12.3514865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:12.3515412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:12.3619978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:12.3620418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:12.7321713Z skip: Need at least 2 CUDA devices (1.911s)
2022-09-27T15:36:12.7366757Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7286
2022-09-27T15:36:12.7396506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7287
2022-09-27T15:36:13.8439945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:13.8440638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:13.8444659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:13.8445348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:13.8497946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:13.8498327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:13.8502122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:13.8502481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:14.1941736Z dist init r=0, world=2
2022-09-27T15:36:14.1968958Z dist init r=1, world=2
2022-09-27T15:36:14.2149565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:14.2150058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:14.2150870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:14.2151730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:36:14.2155220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:14.2155660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:14.5432909Z skip: Need at least 2 CUDA devices (1.811s)
2022-09-27T15:36:14.5479086Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7314
2022-09-27T15:36:15.5994750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:15.5995357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:15.5998405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:15.5998955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:15.9375002Z dist init r=0, world=1
2022-09-27T15:36:15.9383145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:15.9383982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T15:36:15.9387231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:16.2510781Z skip: Need at least 1 CUDA device (1.708s)
2022-09-27T15:36:16.2552026Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7328
2022-09-27T15:36:17.3078545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:17.3079239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:17.3083350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:17.3083988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:17.6463779Z dist init r=0, world=1
2022-09-27T15:36:17.6471717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:17.6472683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T15:36:17.6475726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:17.9582862Z skip: Need at least 1 CUDA device (1.707s)
2022-09-27T15:36:17.9630739Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7342
2022-09-27T15:36:19.0180416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:19.0180790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:19.0183731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:19.0184241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:19.3589005Z dist init r=0, world=1
2022-09-27T15:36:19.3597264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:19.3598043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T15:36:19.3601070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:19.6661781Z skip: Need at least 1 CUDA device (1.708s)
2022-09-27T15:36:19.6662081Z 
2022-09-27T15:36:19.6662506Z ----------------------------------------------------------------------
2022-09-27T15:36:19.6662890Z Ran 44 tests in 77.785s
2022-09-27T15:36:19.6663065Z 
2022-09-27T15:36:19.6663695Z OK (skipped=44)
2022-09-27T15:36:19.6664046Z 
2022-09-27T15:36:19.6664191Z Generating XML reports...
2022-09-27T15:36:19.6739879Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220927153501.xml
2022-09-27T15:36:19.6744493Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220927153501.xml
2022-09-27T15:36:19.9660794Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-09-27 15:36:19.965684]
2022-09-27T15:36:19.9661675Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:36:19.965764]
2022-09-27T15:36:21.1440607Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2022-09-27T15:36:21.1450855Z 
2022-09-27T15:36:21.1450952Z Running tests...
2022-09-27T15:36:21.1451901Z ----------------------------------------------------------------------
2022-09-27T15:36:21.1476538Z   test_apply_activation_checkpointing_wrapper (__main__.CheckpointWrapperTest)
2022-09-27T15:36:21.4319103Z Ensures that `apply_activation_checkpointing_wrapper` can be used ... ok (0.286s)
2022-09-27T15:36:21.4332384Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test requires CUDA (0.001s)
2022-09-27T15:36:21.4400634Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.007s)
2022-09-27T15:36:21.4420208Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2022-09-27T15:36:21.4420705Z Tests that using checkpoint_wrapper or the functional ... skip: Test requires CUDA (0.002s)
2022-09-27T15:36:21.4432345Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-09-27T15:36:21.4441187Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-09-27T15:36:21.4466456Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.002s)
2022-09-27T15:36:21.4466858Z 
2022-09-27T15:36:21.4467351Z ----------------------------------------------------------------------
2022-09-27T15:36:21.4467799Z Ran 7 tests in 0.301s
2022-09-27T15:36:21.4467992Z 
2022-09-27T15:36:21.4468113Z OK (skipped=2)
2022-09-27T15:36:21.4468303Z 
2022-09-27T15:36:21.4468462Z Generating XML reports...
2022-09-27T15:36:21.4499547Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20220927153621.xml
2022-09-27T15:36:21.6937127Z Running distributed/test_distributed_spawn ... [2022-09-27 15:36:21.693356]
2022-09-27T15:36:21.6981589Z MPI not available -- MPI backend tests will be skipped
2022-09-27T15:36:21.7054107Z Running distributed tests for the test backend with env init_method
2022-09-27T15:36:21.7069755Z Running distributed tests for the gloo backend with env init_method
2022-09-27T15:36:21.7075024Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:36:21.707265]
2022-09-27T15:36:21.7083355Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:36:21.708099]
2022-09-27T15:36:23.0963927Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:36:23.0986232Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:36:23.1039682Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T15:36:23.1078186Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1078932Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1079386Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1079840Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1080310Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1080833Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1081348Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1081871Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1082394Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1082993Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1083579Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1084289Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1084856Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1085476Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1085966Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1086453Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1086937Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1087393Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1087821Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1088290Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1088816Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1089339Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1089772Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1090211Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1090668Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1091106Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1091567Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1092049Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1092494Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1092879Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1093299Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1093724Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1094127Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1094532Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1094970Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1095391Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1095841Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1096240Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1096661Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1097142Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1097644Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1098127Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1098571Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1099051Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1099537Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1100010Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1100415Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1100869Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1101345Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1101760Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1102181Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1102744Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1103183Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1103604Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1104264Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1104696Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1105132Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1105621Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1106031Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1106422Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1106815Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1107204Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1107589Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1108013Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1108445Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1108849Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1109216Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1109629Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1110042Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1110431Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1110838Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1111270Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1111670Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1112065Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1112472Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1112882Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1113281Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1113691Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1114099Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1114488Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1114920Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1115361Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1115819Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1116283Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1116761Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1117211Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1117665Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1118137Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1118589Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1119030Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1119473Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1119921Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1120406Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1121008Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1121486Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1121966Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1122465Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1122866Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1123271Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1123657Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1124025Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1124420Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1124820Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1125220Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1125619Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1126040Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1126448Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1126866Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1127284Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1127710Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1128139Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1128569Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1128984Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1129397Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1129809Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1130255Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1130686Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1131091Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1131500Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1131887Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1132263Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1132666Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1133063Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1133456Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1133934Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1134458Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1134941Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1135358Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1135786Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1136237Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1136678Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1137170Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1137624Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1138073Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1138520Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1139044Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1139440Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1139827Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1140309Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1140738Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1141172Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1141638Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1142077Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1142505Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1142977Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1143498Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1144220Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1144876Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1145558Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1146179Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1146846Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1147463Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1148123Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1148791Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1149401Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1149941Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1150425Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1150859Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1151309Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1151742Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1152175Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1152646Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1153126Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1153665Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1154197Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1154665Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1155090Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1155547Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1156043Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1156514Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1157119Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1157590Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1158071Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1158583Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1158995Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1159398Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1159829Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1160284Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1160725Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1161184Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1161664Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1162120Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1162533Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1162952Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1163411Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1163874Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1164251Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1164646Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1165045Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1165412Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1165786Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1166171Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1166573Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1166947Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1167338Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1167717Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1168111Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1168535Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1168933Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1169324Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1169694Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1170087Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1170528Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1170947Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1171438Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1171936Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1172392Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1172864Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1173350Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1173801Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1174247Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1174682Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1175112Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1175523Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1176049Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1176475Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1176966Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1177478Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1177989Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1178491Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1178969Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1179461Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1179938Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1180391Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1180845Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1181306Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1181772Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1182228Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1182714Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1183227Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1183748Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1184367Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1184816Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1185326Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1185749Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1186179Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1186591Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1186996Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1187394Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1187779Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1188150Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1188553Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1188937Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1189338Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1189726Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1190122Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1190546Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1190910Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1191261Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1191628Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1192013Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1192423Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1192828Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1193245Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1193679Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1194041Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1194524Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1194932Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1195353Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1195916Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1196390Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1196821Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1197260Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1197741Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1198167Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1198591Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1199040Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1199483Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1199918Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1200353Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1200792Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1201231Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1201641Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1202084Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1202567Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:23.1203032Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:36:24.1077899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:24.1078292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:24.1086521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:24.1086915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:24.4475620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:24.4485804Z 
2022-09-27T15:36:24.4485958Z Running tests...
2022-09-27T15:36:24.4486304Z ----------------------------------------------------------------------
2022-09-27T15:36:24.6836163Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7406
2022-09-27T15:36:24.6861478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7407
2022-09-27T15:36:24.6888434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7408
2022-09-27T15:36:25.7413551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:25.7414231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:25.7729583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:25.7730034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:25.7741892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:25.7742414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:25.8058614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:25.8059041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:25.8089048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:25.8089441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:25.8400508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:25.8400877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:26.1626738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:26.1627500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:26.1627902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:26.1628512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:26.1629059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:26.1629587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:26.1734103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:26.1734733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:26.2640038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:26.4939011Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:36:26.4939270Z 
2022-09-27T15:36:26.4939743Z ----------------------------------------------------------------------
2022-09-27T15:36:26.4940185Z Ran 1 test in 2.045s
2022-09-27T15:36:26.4940334Z 
2022-09-27T15:36:26.4940408Z OK (skipped=1)
2022-09-27T15:36:26.4940518Z 
2022-09-27T15:36:26.4940619Z Generating XML reports...
2022-09-27T15:36:26.4985611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153624.xml
2022-09-27T15:36:27.7783676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:27.7784566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:27.7792665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:27.7793334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:28.1175428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:28.1185807Z 
2022-09-27T15:36:28.1186101Z Running tests...
2022-09-27T15:36:28.1186744Z ----------------------------------------------------------------------
2022-09-27T15:36:28.1221047Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-09-27T15:36:28.1221487Z 
2022-09-27T15:36:28.1222487Z ----------------------------------------------------------------------
2022-09-27T15:36:28.1222965Z Ran 1 test in 0.004s
2022-09-27T15:36:28.1223091Z 
2022-09-27T15:36:28.1223168Z OK (skipped=1)
2022-09-27T15:36:28.1223288Z 
2022-09-27T15:36:28.1223377Z Generating XML reports...
2022-09-27T15:36:28.1252137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153628.xml
2022-09-27T15:36:29.3267519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:29.3267888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:29.3274792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:29.3275511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:29.6693815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:29.6702828Z 
2022-09-27T15:36:29.6702969Z Running tests...
2022-09-27T15:36:29.6703693Z ----------------------------------------------------------------------
2022-09-27T15:36:29.9045588Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7469
2022-09-27T15:36:29.9070587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7470
2022-09-27T15:36:29.9097764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7471
2022-09-27T15:36:30.9402093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:30.9402482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:30.9689045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:30.9689422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:30.9732375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:30.9733011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:31.0016920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:31.0017596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:31.0046662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:31.0047262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:31.0391889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:31.0392578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:31.3848219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:31.3949615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:31.3950542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:31.3950983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:31.3951683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:31.3952595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:31.3957012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:31.3957812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:31.3958991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:31.7148477Z ok (2.044s)
2022-09-27T15:36:31.7148623Z 
2022-09-27T15:36:31.7148945Z ----------------------------------------------------------------------
2022-09-27T15:36:31.7149199Z Ran 1 test in 2.044s
2022-09-27T15:36:31.7149315Z 
2022-09-27T15:36:31.7149363Z OK
2022-09-27T15:36:31.7149458Z 
2022-09-27T15:36:31.7149552Z Generating XML reports...
2022-09-27T15:36:31.7194061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153629.xml
2022-09-27T15:36:33.0219793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:33.0220187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:33.0227751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:33.0228239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:33.3627089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:33.3636182Z 
2022-09-27T15:36:33.3636308Z Running tests...
2022-09-27T15:36:33.3637030Z ----------------------------------------------------------------------
2022-09-27T15:36:33.5980718Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7522
2022-09-27T15:36:33.6006638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7523
2022-09-27T15:36:33.6034703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7524
2022-09-27T15:36:34.6985481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:34.6986161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:34.7049464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:34.7050097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:34.7227599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:34.7228303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:34.7327271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:34.7327818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:34.7407215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:34.7407789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:34.7569909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:34.7570541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:35.0996149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:35.1079156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:35.1079664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:35.1080287Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:35.1080844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:35.1097307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:35.1186718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:35.1187709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:35.2109602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:35.5085353Z skip: CUDA is not available. (2.144s)
2022-09-27T15:36:35.5085788Z 
2022-09-27T15:36:35.5086788Z ----------------------------------------------------------------------
2022-09-27T15:36:35.5087133Z Ran 1 test in 2.145s
2022-09-27T15:36:35.5087255Z 
2022-09-27T15:36:35.5087336Z OK (skipped=1)
2022-09-27T15:36:35.5087447Z 
2022-09-27T15:36:35.5087714Z Generating XML reports...
2022-09-27T15:36:35.5131643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153633.xml
2022-09-27T15:36:36.8064755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:36.8065115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:36.8073161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:36.8073730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:37.1433597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:37.1443058Z 
2022-09-27T15:36:37.1443359Z Running tests...
2022-09-27T15:36:37.1444057Z ----------------------------------------------------------------------
2022-09-27T15:36:37.3799447Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7575
2022-09-27T15:36:37.3826019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7576
2022-09-27T15:36:37.3852080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7577
2022-09-27T15:36:38.4220357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:38.4220738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:38.4528664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:38.4529133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:38.4811761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:38.4812258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:38.4949021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:38.4949535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:38.5153435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:38.5153959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:38.5288220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:38.5288598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:38.8696761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:38.8731003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:38.8731431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:38.8732058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:38.8732593Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:38.8797882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:38.8838226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:38.8839115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:38.8903674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvk49d02
2022-09-27T15:36:38.8906088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvk49d02/_remote_module_non_scriptable.py
2022-09-27T15:36:38.8909306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lix_3x2
2022-09-27T15:36:38.8911291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lix_3x2/_remote_module_non_scriptable.py
2022-09-27T15:36:38.9806900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:38.9874493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7igruda
2022-09-27T15:36:38.9876034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7igruda/_remote_module_non_scriptable.py
2022-09-27T15:36:39.0046685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:39.0047112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:39.0047465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:39.3905645Z ok (2.246s)
2022-09-27T15:36:39.3905902Z 
2022-09-27T15:36:39.3906424Z ----------------------------------------------------------------------
2022-09-27T15:36:39.3906874Z Ran 1 test in 2.246s
2022-09-27T15:36:39.3907092Z 
2022-09-27T15:36:39.3907198Z OK
2022-09-27T15:36:39.3907364Z 
2022-09-27T15:36:39.3907528Z Generating XML reports...
2022-09-27T15:36:39.3952522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153637.xml
2022-09-27T15:36:40.7136332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:40.7136709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:40.7144563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:40.7145131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:41.0510968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:41.0519950Z 
2022-09-27T15:36:41.0520086Z Running tests...
2022-09-27T15:36:41.0520707Z ----------------------------------------------------------------------
2022-09-27T15:36:41.2887890Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7628
2022-09-27T15:36:41.2914022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7629
2022-09-27T15:36:41.2941366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7630
2022-09-27T15:36:42.3185183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:42.3185667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:42.3494202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:42.3494690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:42.3625146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:42.3625886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:42.3954458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:42.3955077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:42.3975824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:42.3976216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:42.4297053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:42.4297440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:42.7737726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:42.7738573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:42.7739503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:42.7740200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:42.7740740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:42.7741256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:42.7843092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:42.7909299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy60tqps4
2022-09-27T15:36:42.7911860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy60tqps4/_remote_module_non_scriptable.py
2022-09-27T15:36:42.8748358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:42.8749002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:42.8816992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyq20z20
2022-09-27T15:36:42.8818706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzoelyg1
2022-09-27T15:36:42.8819275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyq20z20/_remote_module_non_scriptable.py
2022-09-27T15:36:42.8821166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzoelyg1/_remote_module_non_scriptable.py
2022-09-27T15:36:42.9003801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:42.9004190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:42.9004613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:36:43.2993149Z ok (2.247s)
2022-09-27T15:36:43.2993338Z 
2022-09-27T15:36:43.2993798Z ----------------------------------------------------------------------
2022-09-27T15:36:43.2994227Z Ran 1 test in 2.247s
2022-09-27T15:36:43.2994399Z 
2022-09-27T15:36:43.2994493Z OK
2022-09-27T15:36:43.2994636Z 
2022-09-27T15:36:43.2994770Z Generating XML reports...
2022-09-27T15:36:43.3041003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153641.xml
2022-09-27T15:36:44.6231454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:44.6231842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:44.6239482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:44.6239853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:44.9603479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:44.9612836Z 
2022-09-27T15:36:44.9612955Z Running tests...
2022-09-27T15:36:44.9613437Z ----------------------------------------------------------------------
2022-09-27T15:36:45.1995629Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7681
2022-09-27T15:36:45.2021771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7682
2022-09-27T15:36:45.2048929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7683
2022-09-27T15:36:46.2402924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:46.2403880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:46.2724468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:46.2725111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:46.2973845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:46.2974497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:46.3223077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:46.3223732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:46.3299932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:46.3300542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:46.3568276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:46.3568971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:46.6912103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:46.6912736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:46.6913504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:46.6914126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:46.6914669Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:46.7012660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:46.7020438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:46.7021273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:46.8024750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:47.1100249Z skip: CUDA is not available. (2.148s)
2022-09-27T15:36:47.1100526Z 
2022-09-27T15:36:47.1100985Z ----------------------------------------------------------------------
2022-09-27T15:36:47.1101403Z Ran 1 test in 2.149s
2022-09-27T15:36:47.1101596Z 
2022-09-27T15:36:47.1101720Z OK (skipped=1)
2022-09-27T15:36:47.1101881Z 
2022-09-27T15:36:47.1102047Z Generating XML reports...
2022-09-27T15:36:47.1147758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153644.xml
2022-09-27T15:36:48.4113707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:48.4114370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:48.4122776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:48.4123428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:48.7514645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:48.7524601Z 
2022-09-27T15:36:48.7524878Z Running tests...
2022-09-27T15:36:48.7525345Z ----------------------------------------------------------------------
2022-09-27T15:36:48.9872555Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7734
2022-09-27T15:36:48.9897753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7735
2022-09-27T15:36:48.9924590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7736
2022-09-27T15:36:50.0815950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:50.0816442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:50.0838477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:50.0839156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:50.1165633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:50.1166337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:50.1186652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:50.1187313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:50.1510657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:50.1511373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:50.1828614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:50.1829214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:50.5131324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:50.5232076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:50.5232740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:50.5233381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:50.5233918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:50.5234447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:50.5337999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:50.6244042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:50.6244745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:50.8974399Z skip: CUDA is not available. (2.145s)
2022-09-27T15:36:50.8974688Z 
2022-09-27T15:36:50.8975090Z ----------------------------------------------------------------------
2022-09-27T15:36:50.8975479Z Ran 1 test in 2.145s
2022-09-27T15:36:50.8975666Z 
2022-09-27T15:36:50.8975772Z OK (skipped=1)
2022-09-27T15:36:50.8975931Z 
2022-09-27T15:36:50.8976059Z Generating XML reports...
2022-09-27T15:36:50.9021694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153648.xml
2022-09-27T15:36:52.2531278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:52.2531655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:52.2538559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:52.2538954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:52.5919382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:52.5928858Z 
2022-09-27T15:36:52.5929060Z Running tests...
2022-09-27T15:36:52.5929554Z ----------------------------------------------------------------------
2022-09-27T15:36:52.8288397Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7787
2022-09-27T15:36:52.8314492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7788
2022-09-27T15:36:52.8341401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7789
2022-09-27T15:36:53.9307399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:53.9308087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:53.9461195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:53.9461772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:53.9649540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:53.9649944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:53.9803781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:53.9804393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:53.9882571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:53.9883194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:54.0198391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:54.0198771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:54.3582568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:54.3583003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:54.3583639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:54.3584252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:54.3584771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:54.3585366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:54.3688559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:54.4594986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:54.4595786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:54.7390218Z skip: CUDA is not available. (2.146s)
2022-09-27T15:36:54.7390535Z 
2022-09-27T15:36:54.7391048Z ----------------------------------------------------------------------
2022-09-27T15:36:54.7391301Z Ran 1 test in 2.146s
2022-09-27T15:36:54.7391427Z 
2022-09-27T15:36:54.7391501Z OK (skipped=1)
2022-09-27T15:36:54.7391613Z 
2022-09-27T15:36:54.7391700Z Generating XML reports...
2022-09-27T15:36:54.7435746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153652.xml
2022-09-27T15:36:56.0612912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:56.0613556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:56.0621449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:56.4196159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:56.4196704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:36:56.4205878Z 
2022-09-27T15:36:56.4205979Z Running tests...
2022-09-27T15:36:56.4206386Z ----------------------------------------------------------------------
2022-09-27T15:36:56.6614629Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7840
2022-09-27T15:36:56.6641222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7841
2022-09-27T15:36:56.6668777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7842
2022-09-27T15:36:57.7349677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:57.7350359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:57.7523241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:57.7523832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:57.7679615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:57.7680031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:57.7860717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:57.7861106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:57.7863815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:57.7864309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:57.8178348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:57.8179012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:36:58.1355823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:36:58.1356540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:36:58.1356958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:36:58.1357633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:58.1358537Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:58.1359283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:36:58.1363989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:36:58.1364772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:36:58.2369575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:36:58.4716048Z skip: CUDA is not available. (2.051s)
2022-09-27T15:36:58.4716230Z 
2022-09-27T15:36:58.4716697Z ----------------------------------------------------------------------
2022-09-27T15:36:58.4717135Z Ran 1 test in 2.051s
2022-09-27T15:36:58.4717314Z 
2022-09-27T15:36:58.4717390Z OK (skipped=1)
2022-09-27T15:36:58.4717498Z 
2022-09-27T15:36:58.4717803Z Generating XML reports...
2022-09-27T15:36:58.4764278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153656.xml
2022-09-27T15:36:59.7628786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:36:59.7629166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:36:59.7636624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:36:59.7637044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:00.0993185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:00.1002506Z 
2022-09-27T15:37:00.1002616Z Running tests...
2022-09-27T15:37:00.1003051Z ----------------------------------------------------------------------
2022-09-27T15:37:00.1022445Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T15:37:00.1022804Z 
2022-09-27T15:37:00.1023074Z ----------------------------------------------------------------------
2022-09-27T15:37:00.1023309Z Ran 1 test in 0.002s
2022-09-27T15:37:00.1023429Z 
2022-09-27T15:37:00.1023502Z OK (skipped=1)
2022-09-27T15:37:00.1023612Z 
2022-09-27T15:37:00.1023699Z Generating XML reports...
2022-09-27T15:37:00.1053664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153700.xml
2022-09-27T15:37:01.2915583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:01.2915967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:01.2923788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:01.2924245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:01.6297281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:01.6306363Z 
2022-09-27T15:37:01.6306772Z Running tests...
2022-09-27T15:37:01.6307376Z ----------------------------------------------------------------------
2022-09-27T15:37:01.8668466Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7903
2022-09-27T15:37:01.8696452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7904
2022-09-27T15:37:01.8723093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7905
2022-09-27T15:37:02.9144047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:02.9144691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:02.9437358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:02.9438031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:02.9469837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:02.9470517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:02.9476963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:02.9477598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:02.9795714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:02.9796618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:02.9830826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:02.9831645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:03.2998113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:03.3099422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:03.3100120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:03.3100757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:03.3101280Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:03.3101815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:03.3107936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:03.3108918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:03.4111473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:03.6769198Z skip: CUDA is not available. (2.046s)
2022-09-27T15:37:03.6769708Z 
2022-09-27T15:37:03.6770175Z ----------------------------------------------------------------------
2022-09-27T15:37:03.6770564Z Ran 1 test in 2.046s
2022-09-27T15:37:03.6770756Z 
2022-09-27T15:37:03.6770865Z OK (skipped=1)
2022-09-27T15:37:03.6771040Z 
2022-09-27T15:37:03.6771183Z Generating XML reports...
2022-09-27T15:37:03.6817464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153701.xml
2022-09-27T15:37:04.9723606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:04.9724027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:04.9731688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:04.9732181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:05.3100591Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:05.3110273Z 
2022-09-27T15:37:05.3110403Z Running tests...
2022-09-27T15:37:05.3110985Z ----------------------------------------------------------------------
2022-09-27T15:37:05.5436397Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7956
2022-09-27T15:37:05.5462262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7957
2022-09-27T15:37:05.5489275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7958
2022-09-27T15:37:06.6377909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:06.6690898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:06.6691847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:06.6692394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:06.6909821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:06.6910439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:06.6977358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:06.6978053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:06.7270030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:06.7270694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:06.7323571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:06.7323921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:07.0579387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:07.0580024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:07.0581014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:07.0581567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:07.0582083Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:07.0582605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:07.0685316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:07.0687883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:07.1591864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:07.4538239Z skip: CUDA is not available. (2.142s)
2022-09-27T15:37:07.4538567Z 
2022-09-27T15:37:07.4538947Z ----------------------------------------------------------------------
2022-09-27T15:37:07.4539216Z Ran 1 test in 2.143s
2022-09-27T15:37:07.4539334Z 
2022-09-27T15:37:07.4539408Z OK (skipped=1)
2022-09-27T15:37:07.4539573Z 
2022-09-27T15:37:07.4539653Z Generating XML reports...
2022-09-27T15:37:07.4585206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153705.xml
2022-09-27T15:37:08.8059969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:08.8060344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:08.8068975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:08.8069398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:09.1439795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:09.1448914Z 
2022-09-27T15:37:09.1449051Z Running tests...
2022-09-27T15:37:09.1450195Z ----------------------------------------------------------------------
2022-09-27T15:37:09.3741873Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.229s)
2022-09-27T15:37:09.3742808Z 
2022-09-27T15:37:09.3743027Z ----------------------------------------------------------------------
2022-09-27T15:37:09.3743293Z Ran 1 test in 0.229s
2022-09-27T15:37:09.3743397Z 
2022-09-27T15:37:09.3743473Z OK (skipped=1)
2022-09-27T15:37:09.3743581Z 
2022-09-27T15:37:09.3743666Z Generating XML reports...
2022-09-27T15:37:09.3765369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153709.xml
2022-09-27T15:37:10.6173002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:10.6173634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:10.6181246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:10.6181884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:10.9677425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:10.9686481Z 
2022-09-27T15:37:10.9686590Z Running tests...
2022-09-27T15:37:10.9687041Z ----------------------------------------------------------------------
2022-09-27T15:37:11.2105481Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8019
2022-09-27T15:37:11.2132705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8020
2022-09-27T15:37:11.2159532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8021
2022-09-27T15:37:12.2533402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:12.2568964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:12.2569849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:12.2570470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:12.2879479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:12.2880229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:12.2916973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:12.2917366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:12.3161603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:12.3162007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:12.3492053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:12.3492503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:12.7031736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:12.7133130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:12.7134088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:12.7134713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:12.7135235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:12.7136027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:12.7142754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:12.7143346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:12.7144087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:13.1210868Z ok (2.152s)
2022-09-27T15:37:13.1211115Z 
2022-09-27T15:37:13.1211571Z ----------------------------------------------------------------------
2022-09-27T15:37:13.1211946Z Ran 1 test in 2.152s
2022-09-27T15:37:13.1212377Z 
2022-09-27T15:37:13.1212475Z OK
2022-09-27T15:37:13.1212620Z 
2022-09-27T15:37:13.1212766Z Generating XML reports...
2022-09-27T15:37:13.1258225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153710.xml
2022-09-27T15:37:14.4362642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:14.4363069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:14.4370165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:14.4370565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:14.7757403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:14.7766232Z 
2022-09-27T15:37:14.7766361Z Running tests...
2022-09-27T15:37:14.7766949Z ----------------------------------------------------------------------
2022-09-27T15:37:15.0168445Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8072
2022-09-27T15:37:15.0194681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8073
2022-09-27T15:37:15.0221347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8074
2022-09-27T15:37:16.0531362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:16.0532031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:16.0695019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:16.0695567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:16.0868669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:16.0869273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:16.1038393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:16.1039065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:16.1085793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:16.1086150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:16.1402577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:16.1402954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:16.4564450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:16.4651993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:16.4652440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:16.4653068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:16.4653599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:16.4665660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:16.4758995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:16.4759799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:16.5677962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:16.8267162Z skip: CUDA is not available. (2.050s)
2022-09-27T15:37:16.8267425Z 
2022-09-27T15:37:16.8268075Z ----------------------------------------------------------------------
2022-09-27T15:37:16.8268338Z Ran 1 test in 2.050s
2022-09-27T15:37:16.8268455Z 
2022-09-27T15:37:16.8268556Z OK (skipped=1)
2022-09-27T15:37:16.8268696Z 
2022-09-27T15:37:16.8268830Z Generating XML reports...
2022-09-27T15:37:16.8315591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153714.xml
2022-09-27T15:37:18.1148543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:18.1148903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:18.1156595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:18.1156995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:18.4569997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:18.4578845Z 
2022-09-27T15:37:18.4579389Z Running tests...
2022-09-27T15:37:18.4579813Z ----------------------------------------------------------------------
2022-09-27T15:37:18.6959289Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8125
2022-09-27T15:37:18.6986164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8126
2022-09-27T15:37:18.7012151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8127
2022-09-27T15:37:19.8038102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:19.8038480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:19.8133378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:19.8133728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:19.8225186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:19.8225679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:19.8387316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:19.8387867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:19.8489277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:19.8489956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:19.8572664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:19.8573287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:20.1967397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:20.2068611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:20.2069063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:20.2069680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:20.2070219Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:20.2070903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:20.2076509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:20.2077299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:20.2087391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:20.5058489Z skip: Need at least 3 CUDA devices (2.048s)
2022-09-27T15:37:20.5058721Z 
2022-09-27T15:37:20.5059037Z ----------------------------------------------------------------------
2022-09-27T15:37:20.5059336Z Ran 1 test in 2.048s
2022-09-27T15:37:20.5059454Z 
2022-09-27T15:37:20.5059532Z OK (skipped=1)
2022-09-27T15:37:20.5059647Z 
2022-09-27T15:37:20.5059736Z Generating XML reports...
2022-09-27T15:37:20.5104675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153718.xml
2022-09-27T15:37:21.7922412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:21.7922777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:21.7930929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:21.7931295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:22.1341050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:22.1350839Z 
2022-09-27T15:37:22.1350971Z Running tests...
2022-09-27T15:37:22.1351575Z ----------------------------------------------------------------------
2022-09-27T15:37:22.1369180Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T15:37:22.1369712Z 
2022-09-27T15:37:22.1370173Z ----------------------------------------------------------------------
2022-09-27T15:37:22.1370528Z Ran 1 test in 0.002s
2022-09-27T15:37:22.1370661Z 
2022-09-27T15:37:22.1370750Z OK (skipped=1)
2022-09-27T15:37:22.1370862Z 
2022-09-27T15:37:22.1370951Z Generating XML reports...
2022-09-27T15:37:22.1400629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153722.xml
2022-09-27T15:37:23.3506404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:23.3506789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:23.3515041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:23.3515389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:23.6891845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:23.6901902Z 
2022-09-27T15:37:23.6902193Z Running tests...
2022-09-27T15:37:23.6902815Z ----------------------------------------------------------------------
2022-09-27T15:37:23.6917874Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:37:23.9247969Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8188
2022-09-27T15:37:23.9275085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8189
2022-09-27T15:37:23.9301980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8190
2022-09-27T15:37:24.9969109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:24.9969806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:25.0032513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:25.0033133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:25.0193059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:25.0193846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:25.0318419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:25.0318952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:25.0394613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:25.0395188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:25.0549839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:25.0550461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:25.3990162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:25.4089615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:25.4090307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:25.4091298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:25.4091858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:25.4092389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:25.4198258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:25.4198881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:25.4267732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb3z6dbo
2022-09-27T15:37:25.4268840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnk3342s
2022-09-27T15:37:25.4269704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb3z6dbo/_remote_module_non_scriptable.py
2022-09-27T15:37:25.4271110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnk3342s/_remote_module_non_scriptable.py
2022-09-27T15:37:25.5100341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:25.5170549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw5dv7r_
2022-09-27T15:37:25.5172391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw5dv7r_/_remote_module_non_scriptable.py
2022-09-27T15:37:25.8351683Z ok (2.145s)
2022-09-27T15:37:25.8352104Z 
2022-09-27T15:37:25.8352575Z ----------------------------------------------------------------------
2022-09-27T15:37:25.8352944Z Ran 1 test in 2.145s
2022-09-27T15:37:25.8353143Z 
2022-09-27T15:37:25.8353246Z OK
2022-09-27T15:37:25.8353404Z 
2022-09-27T15:37:25.8353542Z Generating XML reports...
2022-09-27T15:37:25.8400753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153723.xml
2022-09-27T15:37:27.1382602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:27.1382983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:27.1390970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:27.1391492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:27.4775169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:27.4784763Z 
2022-09-27T15:37:27.4784942Z Running tests...
2022-09-27T15:37:27.4785648Z ----------------------------------------------------------------------
2022-09-27T15:37:27.4803305Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:37:27.7143199Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8241
2022-09-27T15:37:27.7169618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8242
2022-09-27T15:37:27.7196735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8243
2022-09-27T15:37:28.7627781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:28.7628464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:28.7765431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:28.7765957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:28.7951029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:28.7951745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:28.7965785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:28.7966170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:28.8122106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:28.8122476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:28.8284756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:28.8285200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:29.1691801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:29.1792474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:29.1793120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:29.1794072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:29.1794631Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:29.1795342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:29.1800703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:29.1801285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:29.1801844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:29.1869596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpib0xcmni
2022-09-27T15:37:29.1871502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpib0xcmni/_remote_module_non_scriptable.py
2022-09-27T15:37:29.1905298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsao1md2k
2022-09-27T15:37:29.1905944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedbvxuc6
2022-09-27T15:37:29.1907970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsao1md2k/_remote_module_non_scriptable.py
2022-09-27T15:37:29.1908622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedbvxuc6/_remote_module_non_scriptable.py
2022-09-27T15:37:29.2199687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:29.2200183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:29.2200632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:29.5243854Z ok (2.045s)
2022-09-27T15:37:29.5244020Z 
2022-09-27T15:37:29.5244351Z ----------------------------------------------------------------------
2022-09-27T15:37:29.5244592Z Ran 1 test in 2.046s
2022-09-27T15:37:29.5244709Z 
2022-09-27T15:37:29.5244772Z OK
2022-09-27T15:37:29.5244864Z 
2022-09-27T15:37:29.5244957Z Generating XML reports...
2022-09-27T15:37:29.5291399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153727.xml
2022-09-27T15:37:30.8083371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:30.8083759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:30.8091351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:30.8091810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:31.1463088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:31.1472512Z 
2022-09-27T15:37:31.1472652Z Running tests...
2022-09-27T15:37:31.1473369Z ----------------------------------------------------------------------
2022-09-27T15:37:31.1492257Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:37:31.3873211Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8294
2022-09-27T15:37:31.3898992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8295
2022-09-27T15:37:31.3925839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8296
2022-09-27T15:37:32.4884744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:32.4885391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:32.5005996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:32.5006626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:32.5231005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:32.5231623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:32.5261693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:32.5262330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:32.5353854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:32.5354496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:32.5586036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:32.5586734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:32.8790603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:32.8792596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:32.8793456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:32.8794063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:32.8794661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:32.8891373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:32.8901295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:32.8970877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36fxd7sx
2022-09-27T15:37:32.8972976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36fxd7sx/_remote_module_non_scriptable.py
2022-09-27T15:37:32.9899345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:32.9902533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:33.0001287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuu964pl
2022-09-27T15:37:33.0003819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuu964pl/_remote_module_non_scriptable.py
2022-09-27T15:37:33.0004530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kfxfo20
2022-09-27T15:37:33.0006365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kfxfo20/_remote_module_non_scriptable.py
2022-09-27T15:37:33.0260142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:33.0260871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:33.0261305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:37:33.2976100Z ok (2.150s)
2022-09-27T15:37:33.2976410Z 
2022-09-27T15:37:33.2976908Z ----------------------------------------------------------------------
2022-09-27T15:37:33.2977177Z Ran 1 test in 2.150s
2022-09-27T15:37:33.2977291Z 
2022-09-27T15:37:33.2977351Z OK
2022-09-27T15:37:33.2977429Z 
2022-09-27T15:37:33.2977524Z Generating XML reports...
2022-09-27T15:37:33.3020797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153731.xml
2022-09-27T15:37:34.6054782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:34.6055149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:34.6061987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:34.6062354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:34.9464877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:34.9474050Z 
2022-09-27T15:37:34.9474179Z Running tests...
2022-09-27T15:37:34.9474827Z ----------------------------------------------------------------------
2022-09-27T15:37:34.9489863Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:37:35.1817031Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8347
2022-09-27T15:37:35.1842714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8348
2022-09-27T15:37:35.1870541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8349
2022-09-27T15:37:36.2611644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:36.2612055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:36.2926819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:36.2927857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:36.2950002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:36.2950617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:36.3071021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:36.3269903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:36.3270463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:36.3270820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:36.3405772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:36.3406132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:36.6591780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:36.6692639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:36.6693204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:36.6693938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:36.6694824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:36.6695483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:36.6701055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:36.6701755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:36.6703668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:36.6771766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0594ejlc
2022-09-27T15:37:36.6773944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0594ejlc/_remote_module_non_scriptable.py
2022-09-27T15:37:36.6802300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7omhen80
2022-09-27T15:37:36.6804734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2cgcper
2022-09-27T15:37:36.6805159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7omhen80/_remote_module_non_scriptable.py
2022-09-27T15:37:36.6806940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2cgcper/_remote_module_non_scriptable.py
2022-09-27T15:37:36.9917617Z ok (2.044s)
2022-09-27T15:37:36.9917853Z 
2022-09-27T15:37:36.9918392Z ----------------------------------------------------------------------
2022-09-27T15:37:36.9918742Z Ran 1 test in 2.044s
2022-09-27T15:37:36.9918846Z 
2022-09-27T15:37:36.9918915Z OK
2022-09-27T15:37:36.9919007Z 
2022-09-27T15:37:36.9919104Z Generating XML reports...
2022-09-27T15:37:36.9963210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153734.xml
2022-09-27T15:37:38.2965700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:38.2966366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:38.2974697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:38.2975324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:38.6352803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:38.6362891Z 
2022-09-27T15:37:38.6363339Z Running tests...
2022-09-27T15:37:38.6363752Z ----------------------------------------------------------------------
2022-09-27T15:37:38.8703994Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8400
2022-09-27T15:37:38.8729527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8401
2022-09-27T15:37:38.8756544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8402
2022-09-27T15:37:39.9296019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:39.9296392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:39.9606573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:39.9607042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:39.9894657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:39.9895338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:40.0213025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:40.0213635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:40.0216175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:40.0216699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:40.0532661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:40.0533334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:40.3705151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:40.3791908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:40.3792619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:40.3793352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:40.3793890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:40.3806024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:40.3899071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:40.3900281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:40.3903767Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.3905139Z STAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4814649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:40.4819056Z STAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4827190Z STAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4827503Z 
2022-09-27T15:37:40.4849610Z STAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4850012Z 
2022-09-27T15:37:40.4851226Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4866130Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4872230Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4873018Z STAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4873312Z 
2022-09-27T15:37:40.4881895Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4886678Z STAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:40.4887039Z 
2022-09-27T15:37:40.4892117Z STAGE:2022-09-27 15:37:40 8402:8402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4892840Z STAGE:2022-09-27 15:37:40 8400:8400 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.4893271Z STAGE:2022-09-27 15:37:40 8401:8401 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:40.8807520Z ok (2.244s)
2022-09-27T15:37:40.8807746Z 
2022-09-27T15:37:40.8808267Z ----------------------------------------------------------------------
2022-09-27T15:37:40.8808640Z Ran 1 test in 2.244s
2022-09-27T15:37:40.8808757Z 
2022-09-27T15:37:40.8808805Z OK
2022-09-27T15:37:40.8808898Z 
2022-09-27T15:37:40.8808996Z Generating XML reports...
2022-09-27T15:37:40.8852559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153738.xml
2022-09-27T15:37:42.1973802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:42.1974179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:42.1981553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:42.1981902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:42.5348311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:42.5357789Z 
2022-09-27T15:37:42.5357938Z Running tests...
2022-09-27T15:37:42.5358548Z ----------------------------------------------------------------------
2022-09-27T15:37:42.7730095Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8456
2022-09-27T15:37:42.7756987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8457
2022-09-27T15:37:42.7784157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8458
2022-09-27T15:37:43.8870246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:43.8870632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:43.9000827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:43.9001332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:43.9200955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:43.9201423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:43.9206585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:43.9207313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:43.9354712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:43.9355318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:43.9540444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:43.9541108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:44.2807773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:44.2908314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:44.2908949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:44.2909718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:44.2910573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:44.2911376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:44.2917651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:44.2918502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:44.2919558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:44.2923431Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2924983Z STAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2925845Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2934037Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2934373Z 
2022-09-27T15:37:44.2934630Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2943328Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2946022Z STAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2946639Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2953727Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2954729Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2955031Z 
2022-09-27T15:37:44.2964216Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2967351Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:44.2967797Z 
2022-09-27T15:37:44.2973722Z STAGE:2022-09-27 15:37:44 8458:8458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2974197Z STAGE:2022-09-27 15:37:44 8457:8457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.2974625Z STAGE:2022-09-27 15:37:44 8456:8456 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:44.6831807Z ok (2.147s)
2022-09-27T15:37:44.6832329Z 
2022-09-27T15:37:44.6832829Z ----------------------------------------------------------------------
2022-09-27T15:37:44.6833110Z Ran 1 test in 2.147s
2022-09-27T15:37:44.6833320Z 
2022-09-27T15:37:44.6833381Z OK
2022-09-27T15:37:44.6833473Z 
2022-09-27T15:37:44.6833565Z Generating XML reports...
2022-09-27T15:37:44.6880410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153742.xml
2022-09-27T15:37:45.9857341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:45.9857715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:45.9865613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:45.9865978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:46.3261253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:46.3271066Z 
2022-09-27T15:37:46.3271170Z Running tests...
2022-09-27T15:37:46.3271751Z ----------------------------------------------------------------------
2022-09-27T15:37:46.5619143Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8512
2022-09-27T15:37:46.5646352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8513
2022-09-27T15:37:46.5673597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8514
2022-09-27T15:37:47.6321584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:47.6322272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:47.6559650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:47.6560297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:47.6646535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:47.6647188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:47.6818214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:47.6818889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:47.6894911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:47.6895451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:47.7148790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:47.7149437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:48.0474501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:48.0475200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:48.0475681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:48.0476330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:48.0476862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:48.0477435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:48.0481997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:48.0482979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:48.0483855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:48.0691239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:37:48.0793384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:37:48.0794128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:37:48.0795015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:48.0795810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:48.0796740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:48.0799477Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0800331Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0800805Z STAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0808774Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0809603Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0810404Z STAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0816866Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0817689Z STAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0818476Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0823540Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0824467Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0825191Z STAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0830890Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0831463Z STAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0831884Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:48.0839241Z STAGE:2022-09-27 15:37:48 8513:8513 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:48 8514:8514 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.0839787Z 
2022-09-27T15:37:48.0840282Z STAGE:2022-09-27 15:37:48 8512:8512 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:48.4727352Z ok (2.145s)
2022-09-27T15:37:48.4727596Z 
2022-09-27T15:37:48.4728011Z ----------------------------------------------------------------------
2022-09-27T15:37:48.4728269Z Ran 1 test in 2.146s
2022-09-27T15:37:48.4728388Z 
2022-09-27T15:37:48.4728437Z OK
2022-09-27T15:37:48.4728565Z 
2022-09-27T15:37:48.4728693Z Generating XML reports...
2022-09-27T15:37:48.4775771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153746.xml
2022-09-27T15:37:49.7806326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:49.7806934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:49.7814034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:49.7814515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:50.1186542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:50.1195913Z 
2022-09-27T15:37:50.1196049Z Running tests...
2022-09-27T15:37:50.1196677Z ----------------------------------------------------------------------
2022-09-27T15:37:50.3552158Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8577
2022-09-27T15:37:50.3578301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8578
2022-09-27T15:37:50.3604916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8579
2022-09-27T15:37:51.4385944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:51.4386631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:51.4432165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:51.4432835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:51.4543120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:51.4543658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:51.4744721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:51.4745409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:51.4793899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:51.4794553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:51.4891470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:51.4892144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:51.8172179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:51.8172900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:51.8173309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:51.8173907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:51.8174450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:51.8174971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:51.8179184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:51.8181179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:51.8181672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:37:51.8182535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:51.8390282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:37:51.8390857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:37:51.8391917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:51.8392855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:51.8394050Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8394790Z STAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8401718Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:51.8402195Z STAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:51.8409326Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8409866Z STAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8413552Z STAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:51.8414064Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:51.8419993Z STAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8420639Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:51.8424274Z STAGE:2022-09-27 15:37:51 8578:8578 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:51 8579:8579 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:51.8424570Z 
2022-09-27T15:37:51.8486101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:37:52.3653392Z ok (2.245s)
2022-09-27T15:37:52.3653623Z 
2022-09-27T15:37:52.3654162Z ----------------------------------------------------------------------
2022-09-27T15:37:52.3654486Z Ran 1 test in 2.246s
2022-09-27T15:37:52.3654602Z 
2022-09-27T15:37:52.3654650Z OK
2022-09-27T15:37:52.3654743Z 
2022-09-27T15:37:52.3654839Z Generating XML reports...
2022-09-27T15:37:52.3699615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153750.xml
2022-09-27T15:37:53.6826602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:53.6826984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:53.6834238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:53.6834589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:54.0202943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:54.0212912Z 
2022-09-27T15:37:54.0213033Z Running tests...
2022-09-27T15:37:54.0213512Z ----------------------------------------------------------------------
2022-09-27T15:37:54.2535872Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8638
2022-09-27T15:37:54.2561383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8639
2022-09-27T15:37:54.2588626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8640
2022-09-27T15:37:55.3617247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:55.3617903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:55.3622570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:55.3623222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:55.3962139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:55.3962610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:55.3968228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:55.3968715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:55.4249309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:55.4249954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:55.4587532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:55.4588120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:55.8028879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:55.8029585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:55.8029981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:55.8030615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:55.8031138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:55.8031670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:55.8135255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:55.8136046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:55.8141083Z STAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.8141599Z STAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9038128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:55.9043083Z STAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9050855Z STAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9051205Z 
2022-09-27T15:37:55.9051505Z STAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9058664Z STAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9059780Z STAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9060308Z 
2022-09-27T15:37:55.9067225Z STAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9067559Z 
2022-09-27T15:37:55.9067822Z STAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9073518Z STAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9074250Z STAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9074838Z STAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:55.9080967Z STAGE:2022-09-27 15:37:55 8639:8639 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9081641Z STAGE:2022-09-27 15:37:55 8638:8638 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:55.9082164Z STAGE:2022-09-27 15:37:55 8640:8640 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:56.2640630Z ok (2.242s)
2022-09-27T15:37:56.2640861Z 
2022-09-27T15:37:56.2641343Z ----------------------------------------------------------------------
2022-09-27T15:37:56.2641766Z Ran 1 test in 2.243s
2022-09-27T15:37:56.2641968Z 
2022-09-27T15:37:56.2642078Z OK
2022-09-27T15:37:56.2642248Z 
2022-09-27T15:37:56.2644148Z Generating XML reports...
2022-09-27T15:37:56.2690266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153754.xml
2022-09-27T15:37:57.5779340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:57.5779714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:57.5787734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:57.5788245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:57.9151856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:37:57.9161092Z 
2022-09-27T15:37:57.9161198Z Running tests...
2022-09-27T15:37:57.9161777Z ----------------------------------------------------------------------
2022-09-27T15:37:58.1509975Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8694
2022-09-27T15:37:58.1535708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8695
2022-09-27T15:37:58.1562820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8696
2022-09-27T15:37:59.2287577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:59.2288174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:59.2608595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:59.2609205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:59.3076926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:59.3077613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:59.3261088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:37:59.3261583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:37:59.3417748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:59.3418166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:59.3593326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:37:59.3593755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:37:59.6866254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:37:59.6965719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:37:59.6966335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:37:59.6967352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:59.6967885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:59.6968500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:37:59.7072922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:37:59.7080060Z STAGE:2022-09-27 15:37:59 8696:8696 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:59.7977255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:37:59.7978004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:37:59.7985926Z STAGE:2022-09-27 15:37:59 8694:8694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:59.7986765Z STAGE:2022-09-27 15:37:59 8695:8695 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:37:59.7995267Z STAGE:2022-09-27 15:37:59 8695:8695 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:37:59 8696:8696 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:37:59.7995749Z 
2022-09-27T15:37:59.7996167Z STAGE:2022-09-27 15:37:59 8694:8694 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:00.1612705Z ok (2.245s)
2022-09-27T15:38:00.1613072Z 
2022-09-27T15:38:00.1613427Z ----------------------------------------------------------------------
2022-09-27T15:38:00.1613683Z Ran 1 test in 2.245s
2022-09-27T15:38:00.1613804Z 
2022-09-27T15:38:00.1613865Z OK
2022-09-27T15:38:00.1613956Z 
2022-09-27T15:38:00.1614051Z Generating XML reports...
2022-09-27T15:38:00.1660516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153757.xml
2022-09-27T15:38:01.4912450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:01.4912835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:01.4920153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:01.4920575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:01.8332989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:01.8341969Z 
2022-09-27T15:38:01.8342108Z Running tests...
2022-09-27T15:38:01.8342727Z ----------------------------------------------------------------------
2022-09-27T15:38:02.0713854Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8750
2022-09-27T15:38:02.0740080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8751
2022-09-27T15:38:02.0766232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8752
2022-09-27T15:38:03.1039217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:03.1039896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:03.1320883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:03.1321583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:03.1369823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:03.1370511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:03.1534786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:03.1535310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:03.1679985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:03.1680482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:03.1871967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:03.1872362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:03.5116972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:03.5218173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:03.5218787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:03.5219604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:03.5220226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:03.5221143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:03.5227188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:03.5227781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:03.5228360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:03.5233180Z STAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5233478Z 
2022-09-27T15:38:03.5233722Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5242070Z STAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5243117Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5243584Z 
2022-09-27T15:38:03.5257704Z STAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5258243Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5261474Z STAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5267813Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5268579Z STAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5269331Z STAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5280882Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5281696Z STAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5282738Z STAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:03.5287808Z STAGE:2022-09-27 15:38:03 8750:8750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5288849Z STAGE:2022-09-27 15:38:03 8751:8751 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:03 8752:8752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:03.5289142Z 
2022-09-27T15:38:03.9812203Z ok (2.147s)
2022-09-27T15:38:03.9812409Z 
2022-09-27T15:38:03.9813171Z ----------------------------------------------------------------------
2022-09-27T15:38:03.9813618Z Ran 1 test in 2.147s
2022-09-27T15:38:03.9813738Z 
2022-09-27T15:38:03.9813872Z OK
2022-09-27T15:38:03.9813970Z 
2022-09-27T15:38:03.9814064Z Generating XML reports...
2022-09-27T15:38:03.9859846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153801.xml
2022-09-27T15:38:05.2869257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:05.2869882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:05.2877718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:05.2878106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:05.6238630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:05.6248025Z 
2022-09-27T15:38:05.6248155Z Running tests...
2022-09-27T15:38:05.6248927Z ----------------------------------------------------------------------
2022-09-27T15:38:05.6264721Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-09-27T15:38:05.6264998Z 
2022-09-27T15:38:05.6265389Z ----------------------------------------------------------------------
2022-09-27T15:38:05.6265837Z Ran 1 test in 0.002s
2022-09-27T15:38:05.6266020Z 
2022-09-27T15:38:05.6266146Z OK (skipped=1)
2022-09-27T15:38:05.6266287Z 
2022-09-27T15:38:05.6266374Z Generating XML reports...
2022-09-27T15:38:05.6295491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153805.xml
2022-09-27T15:38:06.8048553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:06.8048942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:06.8057102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:06.8057480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:07.1430561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:07.1439777Z 
2022-09-27T15:38:07.1440033Z Running tests...
2022-09-27T15:38:07.1440444Z ----------------------------------------------------------------------
2022-09-27T15:38:07.1457158Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-09-27T15:38:07.1457559Z 
2022-09-27T15:38:07.1458042Z ----------------------------------------------------------------------
2022-09-27T15:38:07.1458417Z Ran 1 test in 0.002s
2022-09-27T15:38:07.1458535Z 
2022-09-27T15:38:07.1458610Z OK (skipped=1)
2022-09-27T15:38:07.1458723Z 
2022-09-27T15:38:07.1458822Z Generating XML reports...
2022-09-27T15:38:07.1487642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153807.xml
2022-09-27T15:38:08.3392583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:08.3392965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:08.3400758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:08.3401123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:08.6791376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:08.6801132Z 
2022-09-27T15:38:08.6801563Z Running tests...
2022-09-27T15:38:08.6801960Z ----------------------------------------------------------------------
2022-09-27T15:38:08.9161095Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8826
2022-09-27T15:38:08.9187971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8827
2022-09-27T15:38:08.9215161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8828
2022-09-27T15:38:10.0344601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:10.0345105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:10.0514540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:10.0515121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:10.0522408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:10.0522783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:10.0693366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:10.0693752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:10.0867299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:10.0867673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:10.0880341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:10.0880720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:10.4127214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:10.4171997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:10.4172473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:10.4173106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:10.4173634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:10.4228239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:10.4279571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:10.4280181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:10.5237152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:10.5491546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:10.5492123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:10.5493040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:10.5493757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:10.5494490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:10.5495236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:10.5495997Z STAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5498757Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5505670Z STAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5506133Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5506934Z STAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5507276Z 
2022-09-27T15:38:10.5521025Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5527334Z STAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5527660Z 
2022-09-27T15:38:10.5533107Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5533778Z STAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5534074Z 
2022-09-27T15:38:10.5543103Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5548496Z STAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:10.5548861Z 
2022-09-27T15:38:10.5553121Z STAGE:2022-09-27 15:38:10 8827:8827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5553727Z STAGE:2022-09-27 15:38:10 8826:8826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.5554184Z STAGE:2022-09-27 15:38:10 8828:8828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:10.9265539Z ok (2.246s)
2022-09-27T15:38:10.9265771Z 
2022-09-27T15:38:10.9266116Z ----------------------------------------------------------------------
2022-09-27T15:38:10.9266439Z Ran 1 test in 2.246s
2022-09-27T15:38:10.9266557Z 
2022-09-27T15:38:10.9266620Z OK
2022-09-27T15:38:10.9266712Z 
2022-09-27T15:38:10.9266807Z Generating XML reports...
2022-09-27T15:38:10.9311723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153808.xml
2022-09-27T15:38:12.2742198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:12.2742563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:12.2750629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:12.2751083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:12.6134696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:12.6144420Z 
2022-09-27T15:38:12.6144855Z Running tests...
2022-09-27T15:38:12.6145312Z ----------------------------------------------------------------------
2022-09-27T15:38:12.8493019Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8891
2022-09-27T15:38:12.8519996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8892
2022-09-27T15:38:12.8546937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8893
2022-09-27T15:38:13.8904315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:13.8904954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:13.9120215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:13.9120936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:13.9254239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:13.9255079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:13.9332907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:13.9333448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:13.9489176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:13.9489792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:13.9658945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:13.9659602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:14.2948495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:14.3049696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:14.3050262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:14.3051175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:14.3052043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:14.3052918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:14.3059147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:14.3059693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:14.3264261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:14.3264840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:14.4059355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:14.4062253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:14.4062909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:14.4072618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:14.4073627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:14.4076487Z STAGE:2022-09-27 15:38:14 8892:8892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:14.4077025Z STAGE:2022-09-27 15:38:14 8893:8893 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:14.4084424Z STAGE:2022-09-27 15:38:14 8893:8893 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:14 8892:8892 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:14.4084891Z 
2022-09-27T15:38:14.4097900Z STAGE:2022-09-27 15:38:14 8892:8892 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:14.4098397Z STAGE:2022-09-27 15:38:14 8893:8893 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:14.4103640Z STAGE:2022-09-27 15:38:14 8893:8893 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:14 8892:8892 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:14.4104088Z 
2022-09-27T15:38:14.8599149Z ok (2.245s)
2022-09-27T15:38:14.8599441Z 
2022-09-27T15:38:14.8600130Z ----------------------------------------------------------------------
2022-09-27T15:38:14.8600615Z Ran 1 test in 2.245s
2022-09-27T15:38:14.8600797Z 
2022-09-27T15:38:14.8600847Z OK
2022-09-27T15:38:14.8600940Z 
2022-09-27T15:38:14.8601038Z Generating XML reports...
2022-09-27T15:38:14.8645120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153812.xml
2022-09-27T15:38:16.1880238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:16.1880846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:16.1890847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:16.1891293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:16.5299662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:16.5310155Z 
2022-09-27T15:38:16.5310580Z Running tests...
2022-09-27T15:38:16.5310974Z ----------------------------------------------------------------------
2022-09-27T15:38:16.5327833Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:38:16.5328318Z 
2022-09-27T15:38:16.5328615Z ----------------------------------------------------------------------
2022-09-27T15:38:16.5328871Z Ran 1 test in 0.002s
2022-09-27T15:38:16.5328974Z 
2022-09-27T15:38:16.5329049Z OK (skipped=1)
2022-09-27T15:38:16.5329157Z 
2022-09-27T15:38:16.5329246Z Generating XML reports...
2022-09-27T15:38:16.5358495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153816.xml
2022-09-27T15:38:17.7442433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:17.7443099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:17.7450619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:17.7451281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:18.0827130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:18.0836889Z 
2022-09-27T15:38:18.0837439Z Running tests...
2022-09-27T15:38:18.0837874Z ----------------------------------------------------------------------
2022-09-27T15:38:18.0854831Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:38:18.0855239Z 
2022-09-27T15:38:18.0855567Z ----------------------------------------------------------------------
2022-09-27T15:38:18.0855829Z Ran 1 test in 0.002s
2022-09-27T15:38:18.0855946Z 
2022-09-27T15:38:18.0856022Z OK (skipped=1)
2022-09-27T15:38:18.0856128Z 
2022-09-27T15:38:18.0856220Z Generating XML reports...
2022-09-27T15:38:18.0885922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153818.xml
2022-09-27T15:38:19.2763750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:19.2764142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:19.2771790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:19.2772151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:19.6156360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:19.6166326Z 
2022-09-27T15:38:19.6166721Z Running tests...
2022-09-27T15:38:19.6167133Z ----------------------------------------------------------------------
2022-09-27T15:38:19.8502121Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8972
2022-09-27T15:38:19.8529285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8973
2022-09-27T15:38:19.8556149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8974
2022-09-27T15:38:20.8960682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:20.8961309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:20.9274855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:20.9275486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:20.9331859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:20.9332517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:20.9398982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:20.9399481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:20.9682160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:20.9682789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:20.9751855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:20.9752506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:21.3218655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:21.3319296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:21.3319768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:21.3320489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:21.3321025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:21.3321551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:21.3426611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:21.4330485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:21.4330889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:21.7604712Z ok (2.143s)
2022-09-27T15:38:21.7604965Z 
2022-09-27T15:38:21.7605308Z ----------------------------------------------------------------------
2022-09-27T15:38:21.7605570Z Ran 1 test in 2.144s
2022-09-27T15:38:21.7605690Z 
2022-09-27T15:38:21.7605757Z OK
2022-09-27T15:38:21.7605849Z 
2022-09-27T15:38:21.7605932Z Generating XML reports...
2022-09-27T15:38:21.7652064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153819.xml
2022-09-27T15:38:23.0649840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:23.0650462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:23.0658933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:23.0659593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:23.4066611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:23.4075962Z 
2022-09-27T15:38:23.4076113Z Running tests...
2022-09-27T15:38:23.4076523Z ----------------------------------------------------------------------
2022-09-27T15:38:23.6418704Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9025
2022-09-27T15:38:23.6445819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9026
2022-09-27T15:38:23.6472220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9027
2022-09-27T15:38:24.6700368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:24.6700974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:24.6913107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:24.6913508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:24.7032878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:24.7033232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:24.7242924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:24.7243288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:24.7527364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:24.7527935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:24.7846110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:24.7846626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:25.1272914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:25.1273356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:25.1273804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:25.1274706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:25.1275596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:25.1276163Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:25.1378519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:25.2283831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:25.2284406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:25.2697769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:25.2698223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:25.2797124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:25.2798078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:25.2799001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:25.2799543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:25.3046024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T15:38:25.3047149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-09-27T15:38:25.3047917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:38:25.3048318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T15:38:25.3048875Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:38:25.3049462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:38:25.3372990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T15:38:25.3373586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T15:38:25.3374220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-09-27T15:38:25.3375086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:38:25.3375946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:38:25.3376739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:38:25.7523658Z ok (2.344s)
2022-09-27T15:38:25.7523872Z 
2022-09-27T15:38:25.7524406Z ----------------------------------------------------------------------
2022-09-27T15:38:25.7524829Z Ran 1 test in 2.345s
2022-09-27T15:38:25.7524934Z 
2022-09-27T15:38:25.7525008Z OK
2022-09-27T15:38:25.7525104Z 
2022-09-27T15:38:25.7525198Z Generating XML reports...
2022-09-27T15:38:25.7569687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153823.xml
2022-09-27T15:38:27.0500996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:27.0501373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:27.0509310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:27.0509695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:27.3882772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:27.3892596Z 
2022-09-27T15:38:27.3892728Z Running tests...
2022-09-27T15:38:27.3893294Z ----------------------------------------------------------------------
2022-09-27T15:38:27.3917629Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-09-27T15:38:27.3917954Z 
2022-09-27T15:38:27.3918248Z ----------------------------------------------------------------------
2022-09-27T15:38:27.3918546Z Ran 1 test in 0.002s
2022-09-27T15:38:27.3918648Z 
2022-09-27T15:38:27.3918722Z OK (skipped=1)
2022-09-27T15:38:27.3918829Z 
2022-09-27T15:38:27.3918917Z Generating XML reports...
2022-09-27T15:38:27.3948412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153827.xml
2022-09-27T15:38:28.6044193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:28.6044813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:28.6051651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:28.6052140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:28.9428880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:28.9438283Z 
2022-09-27T15:38:28.9438516Z Running tests...
2022-09-27T15:38:28.9438926Z ----------------------------------------------------------------------
2022-09-27T15:38:29.1790388Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9124
2022-09-27T15:38:29.1816366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9125
2022-09-27T15:38:29.1842738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9126
2022-09-27T15:38:30.3219811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:30.3220182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:30.3328058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:30.3328737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:30.3532478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:30.3532996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:30.3561589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:30.3562134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:30.3685421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:30.3686051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:30.3867586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:30.3868217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:30.7241569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:30.7342650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:30.7343346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:30.7344576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:30.7345490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:30.7346025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:30.7353030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:30.7353660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:30.7354302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:30.7660577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:30.7661103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:30.7661640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:30.7662728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:30.7663654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:30.7664742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:30.7665498Z STAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7665980Z STAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7765305Z STAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7772457Z STAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7773443Z STAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7773891Z 
2022-09-27T15:38:30.7787566Z STAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7787894Z 
2022-09-27T15:38:30.7788162Z STAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7795227Z STAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7796227Z STAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7796721Z 
2022-09-27T15:38:30.7804446Z STAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7805272Z STAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7806056Z STAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:30.7813070Z STAGE:2022-09-27 15:38:30 9126:9126 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7814171Z STAGE:2022-09-27 15:38:30 9124:9124 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:30 9125:9125 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:30.7814706Z 
2022-09-27T15:38:31.1896011Z ok (2.245s)
2022-09-27T15:38:31.1896278Z 
2022-09-27T15:38:31.1896792Z ----------------------------------------------------------------------
2022-09-27T15:38:31.1897168Z Ran 1 test in 2.246s
2022-09-27T15:38:31.1897286Z 
2022-09-27T15:38:31.1897349Z OK
2022-09-27T15:38:31.1897440Z 
2022-09-27T15:38:31.1897551Z Generating XML reports...
2022-09-27T15:38:31.1941598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153828.xml
2022-09-27T15:38:32.5016235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:32.5016619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:32.5024260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:32.5024614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:32.8404318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:32.8414027Z 
2022-09-27T15:38:32.8414107Z Running tests...
2022-09-27T15:38:32.8414572Z ----------------------------------------------------------------------
2022-09-27T15:38:33.0765594Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9189
2022-09-27T15:38:33.0790950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9190
2022-09-27T15:38:33.0817790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9191
2022-09-27T15:38:34.1541482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:34.1542095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:34.1608767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:34.1609420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:34.1882653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:34.1883377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:34.1963775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:34.1964435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:34.2220420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:34.2221023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:34.2541372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:34.2542063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:34.5761143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:34.5862215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:34.5862890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:34.5863763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:34.5864464Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:34.5864991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:34.5970471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:34.6872854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:34.6971106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:34.7177865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:34.7178481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:34.7179103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:34.7179917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:34.7180738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:34.7181625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:34.7182783Z STAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7184543Z STAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7282448Z STAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7290282Z STAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7291143Z STAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7291437Z 
2022-09-27T15:38:34.7303348Z STAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7310167Z STAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7310639Z 
2022-09-27T15:38:34.7317382Z STAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7317857Z 
2022-09-27T15:38:34.7318285Z STAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7325905Z STAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7326693Z STAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7328068Z STAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:34.7335306Z STAGE:2022-09-27 15:38:34 9190:9190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7336006Z STAGE:2022-09-27 15:38:34 9191:9191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:34.7336459Z STAGE:2022-09-27 15:38:34 9189:9189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:35.0868998Z ok (2.245s)
2022-09-27T15:38:35.0869190Z 
2022-09-27T15:38:35.0869521Z ----------------------------------------------------------------------
2022-09-27T15:38:35.0869792Z Ran 1 test in 2.245s
2022-09-27T15:38:35.0869909Z 
2022-09-27T15:38:35.0869970Z OK
2022-09-27T15:38:35.0870064Z 
2022-09-27T15:38:35.0870156Z Generating XML reports...
2022-09-27T15:38:35.0918982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153832.xml
2022-09-27T15:38:36.3536727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:36.3537353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:36.3545385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:36.6941070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:36.6941985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:36.6952080Z 
2022-09-27T15:38:36.6952405Z Running tests...
2022-09-27T15:38:36.6953002Z ----------------------------------------------------------------------
2022-09-27T15:38:36.9303104Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9254
2022-09-27T15:38:36.9328965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9255
2022-09-27T15:38:36.9355645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9256
2022-09-27T15:38:38.0270237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:38.0270853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:38.0581957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:38.0582660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:38.0671802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:38.0672411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:38.0860050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:38.0860690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:38.1010637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:38.1011208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:38.1197754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:38.1198359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:38.4572689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:38.4573302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:38.4573797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:38.4574833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:38.4575402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:38.4575919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:38.4677892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:38.5583487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:38.5584296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:38.5690348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:38.5792177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:38.5792893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:38.5793639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:38.5794182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:38.5794750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:38.5798086Z STAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5798521Z 
2022-09-27T15:38:38.5798904Z STAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5805860Z STAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5806634Z STAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5806984Z 
2022-09-27T15:38:38.5820468Z STAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5825769Z STAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5826214Z 
2022-09-27T15:38:38.5833547Z STAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5834163Z STAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5834603Z STAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5842483Z STAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5846622Z STAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:38.5846904Z 
2022-09-27T15:38:38.5854225Z STAGE:2022-09-27 15:38:38 9256:9256 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:38 9254:9254 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5855218Z STAGE:2022-09-27 15:38:38 9255:9255 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:38.5855540Z 
2022-09-27T15:38:39.0408527Z ok (2.345s)
2022-09-27T15:38:39.0408773Z 
2022-09-27T15:38:39.0409236Z ----------------------------------------------------------------------
2022-09-27T15:38:39.0409629Z Ran 1 test in 2.346s
2022-09-27T15:38:39.0409806Z 
2022-09-27T15:38:39.0409885Z OK
2022-09-27T15:38:39.0410024Z 
2022-09-27T15:38:39.0410161Z Generating XML reports...
2022-09-27T15:38:39.0455735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153836.xml
2022-09-27T15:38:40.3453350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:40.3453978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:40.3462017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:40.3462426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:40.6835515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:40.6844483Z 
2022-09-27T15:38:40.6844615Z Running tests...
2022-09-27T15:38:40.6845230Z ----------------------------------------------------------------------
2022-09-27T15:38:40.9227257Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9319
2022-09-27T15:38:40.9253789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9320
2022-09-27T15:38:40.9280669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9321
2022-09-27T15:38:42.0054168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:42.0054822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:42.0233490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:42.0234230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:42.0250448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:42.0251103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:42.0392597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:42.0393181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:42.0593089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:42.0593841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:42.0606502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:42.0607106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:42.4086482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:42.4086932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:42.4087503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:42.4088454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:42.4089033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:42.4089549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:42.4093608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:42.4094248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:42.4095442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:42.4200972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:42.4300447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:42.4301122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:42.4301826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:42.4302528Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:42.4303318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:42.4307251Z STAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4405784Z STAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4406397Z STAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4415386Z STAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4416139Z STAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4417356Z STAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4433034Z STAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4433352Z 
2022-09-27T15:38:42.4435080Z STAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4442509Z STAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4443261Z STAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4443553Z 
2022-09-27T15:38:42.4455514Z STAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4460806Z STAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:42.4461357Z 
2022-09-27T15:38:42.4468791Z STAGE:2022-09-27 15:38:42 9321:9321 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:42 9320:9320 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:42 9319:9319 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:42.4469499Z 
2022-09-27T15:38:42.4469509Z 
2022-09-27T15:38:42.8328358Z ok (2.148s)
2022-09-27T15:38:42.8328615Z 
2022-09-27T15:38:42.8329436Z ----------------------------------------------------------------------
2022-09-27T15:38:42.8329905Z Ran 1 test in 2.148s
2022-09-27T15:38:42.8330113Z 
2022-09-27T15:38:42.8330170Z OK
2022-09-27T15:38:42.8330276Z 
2022-09-27T15:38:42.8330372Z Generating XML reports...
2022-09-27T15:38:42.8374229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153840.xml
2022-09-27T15:38:44.1456396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:44.1457044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:44.1465660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:44.1466312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:44.4842427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:44.4853335Z 
2022-09-27T15:38:44.4853775Z Running tests...
2022-09-27T15:38:44.4854174Z ----------------------------------------------------------------------
2022-09-27T15:38:44.7205394Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9384
2022-09-27T15:38:44.7231623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9385
2022-09-27T15:38:44.7258764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9386
2022-09-27T15:38:45.7607040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:45.7607764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:45.7928214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:45.7928877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:45.7952024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:45.7952641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:45.8049503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:45.8050109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:45.8296157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:45.8296832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:45.8396144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:45.8396713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:46.1818093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:46.1919283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:46.1920335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:46.1921185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:46.1921818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:46.1922346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:46.1926860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:46.1927863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:46.2132310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:46.2132732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:46.2928256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:46.2930710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:46.2931325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:46.2940808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:46.2941633Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:46.2945170Z STAGE:2022-09-27 15:38:46 9386:9386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:46.2945705Z STAGE:2022-09-27 15:38:46 9385:9385 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:46.2956077Z STAGE:2022-09-27 15:38:46 9386:9386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:46.2956808Z STAGE:2022-09-27 15:38:46 9385:9385 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:46.2967012Z STAGE:2022-09-27 15:38:46 9386:9386 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:46.2968340Z STAGE:2022-09-27 15:38:46 9385:9385 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:46.2973529Z STAGE:2022-09-27 15:38:46 9385:9385 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:46.2974274Z STAGE:2022-09-27 15:38:46 9386:9386 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:46.7311657Z ok (2.245s)
2022-09-27T15:38:46.7311960Z 
2022-09-27T15:38:46.7312497Z ----------------------------------------------------------------------
2022-09-27T15:38:46.7313011Z Ran 1 test in 2.246s
2022-09-27T15:38:46.7313231Z 
2022-09-27T15:38:46.7313350Z OK
2022-09-27T15:38:46.7313506Z 
2022-09-27T15:38:46.7313669Z Generating XML reports...
2022-09-27T15:38:46.7360343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153844.xml
2022-09-27T15:38:48.0346870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:48.0347258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:48.0355155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:48.0355526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:48.3724564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:48.3734121Z 
2022-09-27T15:38:48.3734316Z Running tests...
2022-09-27T15:38:48.3734898Z ----------------------------------------------------------------------
2022-09-27T15:38:48.6086980Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9445
2022-09-27T15:38:48.6112326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9446
2022-09-27T15:38:48.6139190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9447
2022-09-27T15:38:49.6848269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:49.6848926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:49.6899015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:49.6899638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:49.6919532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:49.6920195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:49.7205223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:49.7205869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:49.7257899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:49.7258639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:49.7279126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:49.7279760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:50.0631953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:50.0731067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:50.0731521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:50.0732247Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:50.0732893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:50.0733481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:50.0839676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:50.0852211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:50.0967451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:50.0967867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:50.1742606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:50.1745341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:50.1745970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:50.1776173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:50.1776815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:50.1780764Z STAGE:2022-09-27 15:38:50 9446:9446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:50.1880700Z STAGE:2022-09-27 15:38:50 9447:9447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:50.1888812Z STAGE:2022-09-27 15:38:50 9446:9446 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:50 9447:9447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:50.1889268Z 
2022-09-27T15:38:50.1901686Z STAGE:2022-09-27 15:38:50 9447:9447 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:50 9446:9446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:50.1901992Z 
2022-09-27T15:38:50.1907439Z STAGE:2022-09-27 15:38:50 9446:9446 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:38:50 9447:9447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:50.1907792Z 
2022-09-27T15:38:50.6192293Z ok (2.245s)
2022-09-27T15:38:50.6192469Z 
2022-09-27T15:38:50.6192823Z ----------------------------------------------------------------------
2022-09-27T15:38:50.6193074Z Ran 1 test in 2.246s
2022-09-27T15:38:50.6193192Z 
2022-09-27T15:38:50.6193241Z OK
2022-09-27T15:38:50.6193340Z 
2022-09-27T15:38:50.6193433Z Generating XML reports...
2022-09-27T15:38:50.6237777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153848.xml
2022-09-27T15:38:51.9317180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:51.9317568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:51.9325271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:51.9325633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:52.2705249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:52.2715198Z 
2022-09-27T15:38:52.2715680Z Running tests...
2022-09-27T15:38:52.2716086Z ----------------------------------------------------------------------
2022-09-27T15:38:52.5049021Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9506
2022-09-27T15:38:52.5076037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9507
2022-09-27T15:38:52.5103750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9508
2022-09-27T15:38:53.6079606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:53.6080200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:53.6371706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:53.6372340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:53.6400934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:53.6401514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:53.6484215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:53.6484847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:53.6714645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:53.6715187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:53.6827604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:53.6828218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:54.0201657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:54.0303378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:54.0304272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:54.0304910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:54.0305530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:54.0306045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:54.0408976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:54.1312592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:54.1313789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:54.1314235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:54.1519488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:54.1519976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:54.1520599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:54.1521125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:54.1524543Z STAGE:2022-09-27 15:38:54 9507:9507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:54.1525064Z STAGE:2022-09-27 15:38:54 9508:9508 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:54.1532566Z STAGE:2022-09-27 15:38:54 9508:9508 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:54.1533167Z STAGE:2022-09-27 15:38:54 9507:9507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:54.1546081Z STAGE:2022-09-27 15:38:54 9507:9507 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:54 9508:9508 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:54.1546371Z 
2022-09-27T15:38:54.1551183Z STAGE:2022-09-27 15:38:54 9508:9508 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:54.1551798Z STAGE:2022-09-27 15:38:54 9507:9507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:54.1618623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:54.6155814Z ok (2.344s)
2022-09-27T15:38:54.6156038Z 
2022-09-27T15:38:54.6156356Z ----------------------------------------------------------------------
2022-09-27T15:38:54.6156666Z Ran 1 test in 2.344s
2022-09-27T15:38:54.6156785Z 
2022-09-27T15:38:54.6156833Z OK
2022-09-27T15:38:54.6156925Z 
2022-09-27T15:38:54.6157019Z Generating XML reports...
2022-09-27T15:38:54.6201243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153852.xml
2022-09-27T15:38:55.9518851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:55.9519228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:55.9527073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:55.9527433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:56.2908372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:38:56.2918643Z 
2022-09-27T15:38:56.2919092Z Running tests...
2022-09-27T15:38:56.2919684Z ----------------------------------------------------------------------
2022-09-27T15:38:56.5260961Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9567
2022-09-27T15:38:56.5287842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9568
2022-09-27T15:38:56.5314789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9569
2022-09-27T15:38:57.5949044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:57.5949434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:57.6021283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:57.6021787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:57.6294395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:57.6294773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:57.6375675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:57.6376055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:57.6854402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:57.6854760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:57.7173922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:57.7174525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:38:58.0578622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:38:58.0680048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:38:58.0680901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:58.0681387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:38:58.0681940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:58.0682479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:38:58.0687711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:38:58.0688642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:38:58.0689277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:38:58.0690105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:38:58.0895259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:38:58.0895950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:38:58.0896617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:58.0897156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:58.0901524Z STAGE:2022-09-27 15:38:58 9569:9569 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:58 9568:9568 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:58.0901885Z 
2022-09-27T15:38:58.0910310Z STAGE:2022-09-27 15:38:58 9569:9569 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:58.0910757Z STAGE:2022-09-27 15:38:58 9568:9568 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:58.0934243Z STAGE:2022-09-27 15:38:58 9568:9568 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:38:58 9569:9569 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:38:58.0934532Z 
2022-09-27T15:38:58.0940021Z STAGE:2022-09-27 15:38:58 9568:9568 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:58.0940478Z STAGE:2022-09-27 15:38:58 9569:9569 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:38:58.0994989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:38:58.5363938Z ok (2.244s)
2022-09-27T15:38:58.5364153Z 
2022-09-27T15:38:58.5364603Z ----------------------------------------------------------------------
2022-09-27T15:38:58.5365049Z Ran 1 test in 2.244s
2022-09-27T15:38:58.5365276Z 
2022-09-27T15:38:58.5365376Z OK
2022-09-27T15:38:58.5365557Z 
2022-09-27T15:38:58.5365727Z Generating XML reports...
2022-09-27T15:38:58.5410808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153856.xml
2022-09-27T15:38:59.8407572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:38:59.8407962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:38:59.8415608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:38:59.8416060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:00.1768693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:00.1778781Z 
2022-09-27T15:39:00.1778859Z Running tests...
2022-09-27T15:39:00.1779306Z ----------------------------------------------------------------------
2022-09-27T15:39:00.4134182Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9628
2022-09-27T15:39:00.4161094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9629
2022-09-27T15:39:00.4189304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9630
2022-09-27T15:39:01.5138582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:01.5139265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:01.5345432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:01.5345976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:01.5480169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:01.5480789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:01.5509510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:01.5509958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:01.5699305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:01.5699736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:01.5848935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:01.5849611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:01.9088772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:01.9089511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:01.9089923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:01.9090544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:01.9091081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:01.9091612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:01.9096221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:01.9096727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:01.9097647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:01.9101080Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9101802Z STAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9102279Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9109651Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9110027Z 
2022-09-27T15:39:01.9110298Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9123128Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9123794Z STAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9124231Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9131314Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9132085Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9132842Z STAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9140024Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9140508Z STAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9143488Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:01.9150001Z STAGE:2022-09-27 15:39:01 9629:9629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9150798Z STAGE:2022-09-27 15:39:01 9628:9628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:01.9151761Z STAGE:2022-09-27 15:39:01 9630:9630 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:02.3239016Z ok (2.146s)
2022-09-27T15:39:02.3239238Z 
2022-09-27T15:39:02.3239759Z ----------------------------------------------------------------------
2022-09-27T15:39:02.3240187Z Ran 1 test in 2.146s
2022-09-27T15:39:02.3240292Z 
2022-09-27T15:39:02.3240356Z OK
2022-09-27T15:39:02.3240456Z 
2022-09-27T15:39:02.3240548Z Generating XML reports...
2022-09-27T15:39:02.3286986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153900.xml
2022-09-27T15:39:03.6595784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:03.6596741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:03.6603994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:03.6604642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:03.9975762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:03.9986373Z 
2022-09-27T15:39:03.9986718Z Running tests...
2022-09-27T15:39:03.9987326Z ----------------------------------------------------------------------
2022-09-27T15:39:04.2323825Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9684
2022-09-27T15:39:04.2349120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9685
2022-09-27T15:39:04.2375733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9686
2022-09-27T15:39:05.3307284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:05.3307866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:05.3461014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:05.3461400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:05.3638653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:05.3639168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:05.3666737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:05.3667100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:05.3810133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:05.3810502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:05.3999153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:05.3999568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:05.7385121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:05.7485336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:05.7485828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:05.7486469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:05.7487051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:05.7487911Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:05.7493645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:05.7494428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:05.7494843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:06.0424557Z ok (2.043s)
2022-09-27T15:39:06.0424799Z 
2022-09-27T15:39:06.0425648Z ----------------------------------------------------------------------
2022-09-27T15:39:06.0425918Z Ran 1 test in 2.044s
2022-09-27T15:39:06.0426036Z 
2022-09-27T15:39:06.0426100Z OK
2022-09-27T15:39:06.0426266Z 
2022-09-27T15:39:06.0426371Z Generating XML reports...
2022-09-27T15:39:06.0470899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153903.xml
2022-09-27T15:39:07.3275164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:07.3275553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:07.3282743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:07.3283258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:07.6753259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:07.6761817Z 
2022-09-27T15:39:07.6762140Z Running tests...
2022-09-27T15:39:07.6762788Z ----------------------------------------------------------------------
2022-09-27T15:39:07.9106923Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9737
2022-09-27T15:39:07.9133027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9738
2022-09-27T15:39:07.9159992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9739
2022-09-27T15:39:08.9698911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:08.9699574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:08.9728614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:08.9729292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:08.9966342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:08.9966882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:09.0048819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:09.0049579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:09.0089188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:09.0089806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:09.0306120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:09.0306770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:09.3626552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:09.3727058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:09.3727962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:09.3728623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:09.3729393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:09.3729936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:09.3734563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:09.3736112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:09.3736660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:09.3739573Z STAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3740297Z STAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3742391Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3749816Z STAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3750517Z STAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3751184Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3763422Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3764135Z STAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3764766Z STAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3774051Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3774711Z 
2022-09-27T15:39:09.3774719Z 
2022-09-27T15:39:09.3781763Z STAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3782486Z STAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3783419Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:09.3791891Z STAGE:2022-09-27 15:39:09 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3792612Z STAGE:2022-09-27 15:39:09 9738:9738 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.3793399Z STAGE:2022-09-27 15:39:09 9737:9737 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:09.8211180Z ok (2.145s)
2022-09-27T15:39:09.8211430Z 
2022-09-27T15:39:09.8212114Z ----------------------------------------------------------------------
2022-09-27T15:39:09.8212563Z Ran 1 test in 2.145s
2022-09-27T15:39:09.8212783Z 
2022-09-27T15:39:09.8212900Z OK
2022-09-27T15:39:09.8213008Z 
2022-09-27T15:39:09.8213103Z Generating XML reports...
2022-09-27T15:39:09.8257706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153907.xml
2022-09-27T15:39:11.1211398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:11.1211768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:11.1219470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:11.1219845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:11.4621946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:11.4631849Z 
2022-09-27T15:39:11.4631934Z Running tests...
2022-09-27T15:39:11.4632490Z ----------------------------------------------------------------------
2022-09-27T15:39:11.6979187Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9793
2022-09-27T15:39:11.7005593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9794
2022-09-27T15:39:11.7033838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9795
2022-09-27T15:39:12.7984255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:12.7985171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:12.8122729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:12.8123368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:12.8336931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:12.8337424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:12.8393583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:12.8393940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:12.8467317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:12.8467938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:12.8716527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:12.8717032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:13.1951163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:13.2022275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:13.2022832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:13.2023492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:13.2024193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:13.2052646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:13.2129349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:13.2129957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:13.2134460Z STAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.2134923Z STAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3061209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:13.3066592Z STAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3073333Z STAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3074175Z STAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3074850Z STAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3087674Z STAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3094290Z STAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3094587Z 
2022-09-27T15:39:13.3102559Z STAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3103597Z STAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3104298Z 
2022-09-27T15:39:13.3112717Z STAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3113476Z STAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3114085Z STAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:13.3120269Z STAGE:2022-09-27 15:39:13 9795:9795 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3121392Z STAGE:2022-09-27 15:39:13 9794:9794 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:13 9793:9793 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:13.3121932Z 
2022-09-27T15:39:13.7085616Z ok (2.245s)
2022-09-27T15:39:13.7085889Z 
2022-09-27T15:39:13.7086419Z ----------------------------------------------------------------------
2022-09-27T15:39:13.7086689Z Ran 1 test in 2.245s
2022-09-27T15:39:13.7086807Z 
2022-09-27T15:39:13.7086878Z OK
2022-09-27T15:39:13.7086970Z 
2022-09-27T15:39:13.7087063Z Generating XML reports...
2022-09-27T15:39:13.7132571Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153911.xml
2022-09-27T15:39:15.0264816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:15.0265239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:15.0272974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:15.0273342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:15.3669494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:15.3678966Z 
2022-09-27T15:39:15.3679068Z Running tests...
2022-09-27T15:39:15.3679607Z ----------------------------------------------------------------------
2022-09-27T15:39:15.6016148Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9849
2022-09-27T15:39:15.6041946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9850
2022-09-27T15:39:15.6069319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9851
2022-09-27T15:39:16.7278282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:16.7278942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:16.7404415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:16.7405050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:16.7504816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:16.7505524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:16.7623017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:16.7623551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:16.7763390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:16.7764022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:16.7848771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:16.7849404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:17.1119227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:17.1219414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:17.1220031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:17.1220647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:17.1221182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:17.1221709Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:17.1326001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:17.1331096Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2229981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:17.2230573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:17.2235529Z STAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2235988Z STAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2245771Z STAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2246490Z STAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2246934Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2262265Z STAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2272373Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2272664Z 
2022-09-27T15:39:17.2282491Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2283114Z STAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2283401Z 
2022-09-27T15:39:17.2293996Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2299568Z STAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2300395Z STAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:17.2307732Z STAGE:2022-09-27 15:39:17 9851:9851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2308219Z STAGE:2022-09-27 15:39:17 9850:9850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.2308662Z STAGE:2022-09-27 15:39:17 9849:9849 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:17.6120426Z ok (2.244s)
2022-09-27T15:39:17.6120602Z 
2022-09-27T15:39:17.6120987Z ----------------------------------------------------------------------
2022-09-27T15:39:17.6121243Z Ran 1 test in 2.244s
2022-09-27T15:39:17.6121362Z 
2022-09-27T15:39:17.6121410Z OK
2022-09-27T15:39:17.6121502Z 
2022-09-27T15:39:17.6121598Z Generating XML reports...
2022-09-27T15:39:17.6165536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153915.xml
2022-09-27T15:39:18.9166889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:18.9167275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:18.9174233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:18.9174660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:19.2593219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:19.2602393Z 
2022-09-27T15:39:19.2602471Z Running tests...
2022-09-27T15:39:19.2603489Z ----------------------------------------------------------------------
2022-09-27T15:39:19.4936026Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9905
2022-09-27T15:39:19.4962061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9906
2022-09-27T15:39:19.4989668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9907
2022-09-27T15:39:20.5994758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:20.5995371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:20.6019537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:20.6019917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:20.6042239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:20.6042578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:20.6348633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:20.6349051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:20.6383394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:20.6384022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:20.6401783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:20.6402440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:20.9782179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:20.9882888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:20.9883594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:20.9884610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:20.9885147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:20.9885667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:20.9891675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:20.9892349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:20.9893296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:21.3035169Z ok (2.043s)
2022-09-27T15:39:21.3035486Z 
2022-09-27T15:39:21.3035923Z ----------------------------------------------------------------------
2022-09-27T15:39:21.3036276Z Ran 1 test in 2.043s
2022-09-27T15:39:21.3036417Z 
2022-09-27T15:39:21.3036506Z OK
2022-09-27T15:39:21.3036599Z 
2022-09-27T15:39:21.3036884Z Generating XML reports...
2022-09-27T15:39:21.3081611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153919.xml
2022-09-27T15:39:22.5948028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:22.5948415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:22.5956261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:22.5956724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:22.9350102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:22.9358800Z 
2022-09-27T15:39:22.9358911Z Running tests...
2022-09-27T15:39:22.9359512Z ----------------------------------------------------------------------
2022-09-27T15:39:23.1789676Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9958
2022-09-27T15:39:23.1815019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9959
2022-09-27T15:39:23.1841446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9960
2022-09-27T15:39:24.2625954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:24.2626550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:24.2942943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:24.2943672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:24.2944734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:24.2945447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:24.2946781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:24.2947404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:24.3298466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:24.3299072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:24.3300135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:24.3300709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:24.6755922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:24.6857549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:24.6858658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:24.6859396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:24.6860030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:24.6860553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:24.6867227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:24.6867905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:24.6868260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:24.6973998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:24.7073022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:24.7073608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:24.7074250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:24.7075021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:24.7075798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:24.7079052Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7177875Z STAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7178333Z STAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7183817Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7184444Z STAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7184903Z STAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7191685Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7192823Z STAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7193318Z 
2022-09-27T15:39:24.7198478Z STAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7199329Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7199630Z 
2022-09-27T15:39:24.7204881Z STAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7206404Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:24.7206854Z 
2022-09-27T15:39:24.7212952Z STAGE:2022-09-27 15:39:24 9959:9959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7213622Z STAGE:2022-09-27 15:39:24 9958:9958 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:24.7214059Z STAGE:2022-09-27 15:39:24 9960:9960 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:25.0893206Z ok (2.153s)
2022-09-27T15:39:25.0893502Z 
2022-09-27T15:39:25.0893972Z ----------------------------------------------------------------------
2022-09-27T15:39:25.0894229Z Ran 1 test in 2.153s
2022-09-27T15:39:25.0894347Z 
2022-09-27T15:39:25.0894408Z OK
2022-09-27T15:39:25.0894500Z 
2022-09-27T15:39:25.0894596Z Generating XML reports...
2022-09-27T15:39:25.0939031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153922.xml
2022-09-27T15:39:26.4225839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:26.4226230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:26.4233890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:26.4234522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:26.7616197Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:26.7625733Z 
2022-09-27T15:39:26.7626104Z Running tests...
2022-09-27T15:39:26.7626461Z ----------------------------------------------------------------------
2022-09-27T15:39:26.9970287Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10023
2022-09-27T15:39:26.9996621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10024
2022-09-27T15:39:27.0023269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10025
2022-09-27T15:39:28.0325741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:28.0326158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:28.0422476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:28.0604324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:28.0605147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:28.0605628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:28.0663689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:28.0664399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:28.0780533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:28.0781129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:28.0949985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:28.0950605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:28.4363599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:28.4465362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:28.4465978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:28.4466898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:28.4467639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:28.4468162Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:28.4473581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:28.4474037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:28.4476265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:28.4581461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:28.4682552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:28.4683126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:28.4683689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:28.4684213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:28.4684943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:28.4687703Z STAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4688258Z STAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4787960Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4793596Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4793964Z 
2022-09-27T15:39:28.4794231Z STAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4800644Z STAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4800971Z 
2022-09-27T15:39:28.4801359Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4808126Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4809140Z STAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4809645Z 
2022-09-27T15:39:28.4814166Z STAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4814818Z STAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4815469Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:28.4821232Z STAGE:2022-09-27 15:39:28 10024:10024 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4821732Z STAGE:2022-09-27 15:39:28 10025:10025 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.4822223Z STAGE:2022-09-27 15:39:28 10023:10023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:28.9071822Z ok (2.144s)
2022-09-27T15:39:28.9072014Z 
2022-09-27T15:39:28.9072455Z ----------------------------------------------------------------------
2022-09-27T15:39:28.9072906Z Ran 1 test in 2.145s
2022-09-27T15:39:28.9073131Z 
2022-09-27T15:39:28.9073260Z OK
2022-09-27T15:39:28.9073403Z 
2022-09-27T15:39:28.9073484Z Generating XML reports...
2022-09-27T15:39:28.9118611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153926.xml
2022-09-27T15:39:30.2420360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:30.2420803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:30.2428708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:30.2429081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:30.5798704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:30.5808471Z 
2022-09-27T15:39:30.5808573Z Running tests...
2022-09-27T15:39:30.5809043Z ----------------------------------------------------------------------
2022-09-27T15:39:30.8169401Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10088
2022-09-27T15:39:30.8196645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10089
2022-09-27T15:39:30.8224843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10090
2022-09-27T15:39:31.8978813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:31.8979720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:31.9295530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:31.9296168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:31.9308257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:31.9308847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:31.9436439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:31.9437107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:31.9653450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:31.9654014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:31.9779686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:31.9780363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:32.3122796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:32.3224485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:32.3225366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:32.3226062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:32.3226584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:32.3227115Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:32.3234099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:32.3235185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:32.3235788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:32.3340459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:32.3442188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:32.3442696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:32.3443675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:32.3444222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:32.3444743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:32.3447033Z STAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3447318Z 
2022-09-27T15:39:32.3547080Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3552754Z STAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3553794Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3554375Z 
2022-09-27T15:39:32.3560968Z STAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3561628Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3562056Z STAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3568528Z STAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3569291Z STAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3570077Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3574596Z STAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3575091Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3575690Z STAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:32.3582053Z STAGE:2022-09-27 15:39:32 10090:10090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3582753Z STAGE:2022-09-27 15:39:32 10089:10089 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:32 10088:10088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:32.3583054Z 
2022-09-27T15:39:32.7276363Z ok (2.146s)
2022-09-27T15:39:32.7276604Z 
2022-09-27T15:39:32.7277134Z ----------------------------------------------------------------------
2022-09-27T15:39:32.7277419Z Ran 1 test in 2.147s
2022-09-27T15:39:32.7277547Z 
2022-09-27T15:39:32.7277608Z OK
2022-09-27T15:39:32.7277698Z 
2022-09-27T15:39:32.7277790Z Generating XML reports...
2022-09-27T15:39:32.7323224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153930.xml
2022-09-27T15:39:34.0555118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:34.0555484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:34.0563037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:34.0563462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:34.3952308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:34.3961401Z 
2022-09-27T15:39:34.3961533Z Running tests...
2022-09-27T15:39:34.3962120Z ----------------------------------------------------------------------
2022-09-27T15:39:34.6334011Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10153
2022-09-27T15:39:34.6361046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10154
2022-09-27T15:39:34.6388295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10155
2022-09-27T15:39:35.7195526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:35.7196178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:35.7288559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:35.7289236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:35.7538018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:35.7538715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:35.7581622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:35.7581996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:35.7633695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:35.7634262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:35.7905902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:35.7906366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:36.1133759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:36.1134378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:36.1134758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:36.1135394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:36.1135928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:36.1136443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:36.1239916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:36.2142740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:36.2143413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:36.2249871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:36.2352246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:36.2352886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:36.2353735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:36.2354540Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:36.2355402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:36.2357366Z STAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2358354Z STAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2358803Z 
2022-09-27T15:39:36.2364560Z STAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.2364875Z 
2022-09-27T15:39:36.2365163Z STAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.2373198Z STAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2373624Z 
2022-09-27T15:39:36.2373795Z 
2022-09-27T15:39:36.2380167Z STAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.2380807Z 
2022-09-27T15:39:36.2381217Z STAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.2386091Z STAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2386814Z STAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2387334Z STAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:36.2395584Z STAGE:2022-09-27 15:39:36 10154:10154 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:36 10153:10153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.2396095Z 
2022-09-27T15:39:36.2396566Z STAGE:2022-09-27 15:39:36 10155:10155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:36.6440831Z ok (2.248s)
2022-09-27T15:39:36.6441085Z 
2022-09-27T15:39:36.6441592Z ----------------------------------------------------------------------
2022-09-27T15:39:36.6442068Z Ran 1 test in 2.248s
2022-09-27T15:39:36.6442259Z 
2022-09-27T15:39:36.6442369Z OK
2022-09-27T15:39:36.6442534Z 
2022-09-27T15:39:36.6442694Z Generating XML reports...
2022-09-27T15:39:36.6487918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153934.xml
2022-09-27T15:39:37.9491233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:37.9491675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:37.9499472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:37.9499839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:38.2879203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:38.2889572Z 
2022-09-27T15:39:38.2889822Z Running tests...
2022-09-27T15:39:38.2890506Z ----------------------------------------------------------------------
2022-09-27T15:39:38.5224635Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10218
2022-09-27T15:39:38.5251734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10219
2022-09-27T15:39:38.5278904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10220
2022-09-27T15:39:39.6426666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:39.6427377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:39.6470575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:39.6471058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:39.6563343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:39.6563682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:39.6796733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:39.6797124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:39.6843524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:39.6844030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:39.6930665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:39.6931127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:40.0287186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:40.0388384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:40.0388887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:40.0389505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:40.0390150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:40.0391044Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:40.0395947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:40.0396798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:40.0397687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:40.0398256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:40.0602924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:40.0603664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:40.0604283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:40.0604869Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:40.0606727Z STAGE:2022-09-27 15:39:40 10219:10219 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:40.0607553Z STAGE:2022-09-27 15:39:40 10220:10220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:40.0613806Z STAGE:2022-09-27 15:39:40 10220:10220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:40.0614364Z STAGE:2022-09-27 15:39:40 10219:10219 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:40.0621416Z STAGE:2022-09-27 15:39:40 10220:10220 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:40 10219:10219 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:40.0621722Z 
2022-09-27T15:39:40.0626354Z STAGE:2022-09-27 15:39:40 10220:10220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:40.0627024Z STAGE:2022-09-27 15:39:40 10219:10219 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:40.0703069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:40.5328389Z ok (2.243s)
2022-09-27T15:39:40.5328626Z 
2022-09-27T15:39:40.5329149Z ----------------------------------------------------------------------
2022-09-27T15:39:40.5329490Z Ran 1 test in 2.244s
2022-09-27T15:39:40.5329594Z 
2022-09-27T15:39:40.5329653Z OK
2022-09-27T15:39:40.5329743Z 
2022-09-27T15:39:40.5329836Z Generating XML reports...
2022-09-27T15:39:40.5375075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153938.xml
2022-09-27T15:39:41.8409144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:41.8409663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:41.8417400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:41.8417851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:42.1769172Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:42.1779117Z 
2022-09-27T15:39:42.1779386Z Running tests...
2022-09-27T15:39:42.1779983Z ----------------------------------------------------------------------
2022-09-27T15:39:42.4130019Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10279
2022-09-27T15:39:42.4156736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10280
2022-09-27T15:39:42.4183240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10281
2022-09-27T15:39:43.4822806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:43.5062203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:43.5063215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:43.5063968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:43.5092998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:43.5093620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:43.5151703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:43.5152231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:43.5416168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:43.5416889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:43.5440449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:43.5441060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:43.8730578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:43.8830872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:43.8831852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:43.8832894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:43.8834049Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:43.8835162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:43.8839150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:43.8839701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:43.8840424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:43.8840866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:43.8946719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:43.8947432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:43.8948205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:43.8948746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:43.8950772Z STAGE:2022-09-27 15:39:43 10280:10280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:43.8952676Z STAGE:2022-09-27 15:39:43 10281:10281 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:43.8958546Z STAGE:2022-09-27 15:39:43 10281:10281 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:43 10280:10280 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:43.8958860Z 
2022-09-27T15:39:43.8965593Z STAGE:2022-09-27 15:39:43 10280:10280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:43.8966203Z STAGE:2022-09-27 15:39:43 10281:10281 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:43.8970099Z STAGE:2022-09-27 15:39:43 10280:10280 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:43.8970553Z STAGE:2022-09-27 15:39:43 10281:10281 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:43.9044418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:44.3230411Z ok (2.145s)
2022-09-27T15:39:44.3230632Z 
2022-09-27T15:39:44.3231129Z ----------------------------------------------------------------------
2022-09-27T15:39:44.3231394Z Ran 1 test in 2.145s
2022-09-27T15:39:44.3231503Z 
2022-09-27T15:39:44.3231566Z OK
2022-09-27T15:39:44.3231659Z 
2022-09-27T15:39:44.3231754Z Generating XML reports...
2022-09-27T15:39:44.3276841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153942.xml
2022-09-27T15:39:45.6450549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:45.6451194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:45.6459587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:45.6460237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:45.9840373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:45.9849497Z 
2022-09-27T15:39:45.9849878Z Running tests...
2022-09-27T15:39:45.9850284Z ----------------------------------------------------------------------
2022-09-27T15:39:46.2197819Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10340
2022-09-27T15:39:46.2223983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10341
2022-09-27T15:39:46.2251107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10342
2022-09-27T15:39:47.2536884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:47.2537544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:47.2574192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:47.2574797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:47.2750923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:47.2751599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:47.2889851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:47.2890380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:47.2932387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:47.2933019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:47.3101578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:47.3102200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:47.6462985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:47.6563482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:47.6564118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:47.6565019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:47.6565548Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:47.6566075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:47.6571486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:47.6572731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:47.6573429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:47.6573855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:47.6678223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:47.6678763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:47.6679347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:47.6679925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:47.6682118Z STAGE:2022-09-27 15:39:47 10341:10341 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:47.6776757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:47.6782781Z STAGE:2022-09-27 15:39:47 10342:10342 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:47.6789453Z STAGE:2022-09-27 15:39:47 10342:10342 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:47.6790025Z STAGE:2022-09-27 15:39:47 10341:10341 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:47.6796389Z STAGE:2022-09-27 15:39:47 10342:10342 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:47.6797075Z STAGE:2022-09-27 15:39:47 10341:10341 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:47.6801904Z STAGE:2022-09-27 15:39:47 10341:10341 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:47 10342:10342 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:47.6802207Z 
2022-09-27T15:39:48.1301996Z ok (2.145s)
2022-09-27T15:39:48.1302227Z 
2022-09-27T15:39:48.1302711Z ----------------------------------------------------------------------
2022-09-27T15:39:48.1303161Z Ran 1 test in 2.145s
2022-09-27T15:39:48.1303357Z 
2022-09-27T15:39:48.1303435Z OK
2022-09-27T15:39:48.1303528Z 
2022-09-27T15:39:48.1304940Z Generating XML reports...
2022-09-27T15:39:48.1353906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153945.xml
2022-09-27T15:39:49.4449705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:49.4450364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:49.4458447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:49.4459110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:49.7814202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:49.7824356Z 
2022-09-27T15:39:49.7824729Z Running tests...
2022-09-27T15:39:49.7825417Z ----------------------------------------------------------------------
2022-09-27T15:39:50.0192545Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10401
2022-09-27T15:39:50.0217367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10402
2022-09-27T15:39:50.0245973Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10403
2022-09-27T15:39:51.0516138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:51.0516739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:51.0645588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:51.0646282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:51.0866816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:51.0867707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:51.0872130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:51.0872776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:51.1027030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:51.1027703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:51.1217813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:51.1218450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:51.4586580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:51.4687437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:51.4688203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:51.4689317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:51.4690299Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:51.4691020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:51.4695077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:51.4696035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:51.4696689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:51.4697141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:39:51.4901488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:39:51.4902320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:39:51.4903201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:51.4904425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:51.4906459Z STAGE:2022-09-27 15:39:51 10402:10402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:51.5000801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:39:51.5005949Z STAGE:2022-09-27 15:39:51 10403:10403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:51.5012388Z STAGE:2022-09-27 15:39:51 10403:10403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:51.5012885Z STAGE:2022-09-27 15:39:51 10402:10402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:51.5019552Z STAGE:2022-09-27 15:39:51 10402:10402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:51.5020110Z STAGE:2022-09-27 15:39:51 10403:10403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:51.5024705Z STAGE:2022-09-27 15:39:51 10403:10403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:51.5025220Z STAGE:2022-09-27 15:39:51 10402:10402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:51.9296064Z ok (2.147s)
2022-09-27T15:39:51.9296320Z 
2022-09-27T15:39:51.9296869Z ----------------------------------------------------------------------
2022-09-27T15:39:51.9297330Z Ran 1 test in 2.147s
2022-09-27T15:39:51.9297537Z 
2022-09-27T15:39:51.9297625Z OK
2022-09-27T15:39:51.9297723Z 
2022-09-27T15:39:51.9297824Z Generating XML reports...
2022-09-27T15:39:51.9342729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153949.xml
2022-09-27T15:39:53.2875760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:53.2876168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:53.2883438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:53.2883975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:53.6278246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:53.6287536Z 
2022-09-27T15:39:53.6287910Z Running tests...
2022-09-27T15:39:53.6288496Z ----------------------------------------------------------------------
2022-09-27T15:39:53.8616315Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10462
2022-09-27T15:39:53.8642924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10463
2022-09-27T15:39:53.8669754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10464
2022-09-27T15:39:54.8998158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:54.8998833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:54.9212742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:54.9213262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:54.9323461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:54.9324082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:54.9437434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:54.9437986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:54.9568582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:54.9569219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:54.9768730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:54.9769354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:55.3041249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:55.3142649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:55.3143393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:55.3144333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:55.3145017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:55.3147373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:55.3151266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:55.3153197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:55.3155860Z STAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.3157996Z STAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4152228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:55.4157103Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4162429Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.4163747Z STAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.4164194Z 
2022-09-27T15:39:55.4171421Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4173738Z STAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4174249Z STAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4181769Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.4182463Z STAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.4182764Z 
2022-09-27T15:39:55.4187544Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4189745Z STAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:55.4190190Z 
2022-09-27T15:39:55.4197309Z STAGE:2022-09-27 15:39:55 10463:10463 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:55 10464:10464 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.4197810Z 
2022-09-27T15:39:55.4198507Z STAGE:2022-09-27 15:39:55 10462:10462 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:55.7719225Z ok (2.143s)
2022-09-27T15:39:55.7719757Z 
2022-09-27T15:39:55.7720135Z ----------------------------------------------------------------------
2022-09-27T15:39:55.7720375Z Ran 1 test in 2.143s
2022-09-27T15:39:55.7720493Z 
2022-09-27T15:39:55.7720555Z OK
2022-09-27T15:39:55.7720648Z 
2022-09-27T15:39:55.7720743Z Generating XML reports...
2022-09-27T15:39:55.7764610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153953.xml
2022-09-27T15:39:57.0915160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:57.0915824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:57.0923987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:57.0924629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:57.4325652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:39:57.4335760Z 
2022-09-27T15:39:57.4336271Z Running tests...
2022-09-27T15:39:57.4336687Z ----------------------------------------------------------------------
2022-09-27T15:39:57.6674499Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10518
2022-09-27T15:39:57.6701288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10519
2022-09-27T15:39:57.6728136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10520
2022-09-27T15:39:58.7182432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:58.7183092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:58.7513564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:58.7514237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:58.7594796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:58.7595187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:58.7620170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:39:58.7620619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:39:58.7947913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:58.7972386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:58.7972952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:39:58.7973321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:39:59.1475085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:39:59.1575993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:39:59.1576617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:39:59.1577403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:59.1577938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:59.1578625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:39:59.1586167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:39:59.1590926Z STAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2586006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:39:59.2587074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:39:59.2591811Z STAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2592337Z STAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2599546Z STAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.2600307Z STAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.2601038Z STAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.2607578Z STAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2609637Z STAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2610104Z 
2022-09-27T15:39:59.2617780Z STAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.2618513Z 
2022-09-27T15:39:59.2618519Z 
2022-09-27T15:39:59.2622874Z STAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2623643Z STAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2624284Z STAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:39:59.2630759Z STAGE:2022-09-27 15:39:59 10519:10519 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:39:59 10520:10520 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.2631296Z 
2022-09-27T15:39:59.2631752Z STAGE:2022-09-27 15:39:59 10518:10518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:39:59.6779524Z ok (2.244s)
2022-09-27T15:39:59.6779836Z 
2022-09-27T15:39:59.6780322Z ----------------------------------------------------------------------
2022-09-27T15:39:59.6780757Z Ran 1 test in 2.244s
2022-09-27T15:39:59.6780982Z 
2022-09-27T15:39:59.6781100Z OK
2022-09-27T15:39:59.6781243Z 
2022-09-27T15:39:59.6781367Z Generating XML reports...
2022-09-27T15:39:59.6827593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153957.xml
2022-09-27T15:40:00.9757869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:00.9758496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:00.9766773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:00.9767412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:01.3202783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:01.3213320Z 
2022-09-27T15:40:01.3213732Z Running tests...
2022-09-27T15:40:01.3214398Z ----------------------------------------------------------------------
2022-09-27T15:40:01.5597472Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10574
2022-09-27T15:40:01.5622629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10575
2022-09-27T15:40:01.5649373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10576
2022-09-27T15:40:02.6418768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:02.6419369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:02.6755328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:02.6755971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:02.6918951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:02.6919610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:02.7238626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:02.7239138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:02.7239600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:02.7239949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:02.7575371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:02.7575897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:03.0723557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:03.0824832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:03.0825809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:03.0826212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:03.0826742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:03.0827269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:03.0832579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:03.0833908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:03.1838525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:03.4701457Z skip: CUDA is not available. (2.148s)
2022-09-27T15:40:03.4701667Z 
2022-09-27T15:40:03.4702062Z ----------------------------------------------------------------------
2022-09-27T15:40:03.4702382Z Ran 1 test in 2.149s
2022-09-27T15:40:03.4702504Z 
2022-09-27T15:40:03.4702564Z OK (skipped=1)
2022-09-27T15:40:03.4702676Z 
2022-09-27T15:40:03.4702761Z Generating XML reports...
2022-09-27T15:40:03.4749281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154001.xml
2022-09-27T15:40:04.7790369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:04.7790736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:04.7798290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:04.7798652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:05.1152534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:05.1161338Z 
2022-09-27T15:40:05.1161479Z Running tests...
2022-09-27T15:40:05.1162779Z ----------------------------------------------------------------------
2022-09-27T15:40:05.3514317Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10627
2022-09-27T15:40:05.3539491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10628
2022-09-27T15:40:05.3566951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10629
2022-09-27T15:40:06.4528091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:06.4528466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:06.4618121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:06.4618645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:06.4874834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:06.4875298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:06.5029864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:06.5030224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:06.5093239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:06.5093878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:06.5367525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:06.5368192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:06.8623564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:06.8624214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:06.8624581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:06.8625258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:06.8625791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:06.8626319Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:06.8730032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:06.9637559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:06.9637988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:07.2618865Z skip: CUDA is not available. (2.145s)
2022-09-27T15:40:07.2619155Z 
2022-09-27T15:40:07.2619659Z ----------------------------------------------------------------------
2022-09-27T15:40:07.2619977Z Ran 1 test in 2.146s
2022-09-27T15:40:07.2620091Z 
2022-09-27T15:40:07.2620162Z OK (skipped=1)
2022-09-27T15:40:07.2620270Z 
2022-09-27T15:40:07.2620356Z Generating XML reports...
2022-09-27T15:40:07.2664959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154005.xml
2022-09-27T15:40:08.5611014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:08.5611620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:08.5619215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:08.5619825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:08.9004312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:08.9013643Z 
2022-09-27T15:40:08.9013777Z Running tests...
2022-09-27T15:40:08.9014351Z ----------------------------------------------------------------------
2022-09-27T15:40:09.1385528Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10680
2022-09-27T15:40:09.1412117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10681
2022-09-27T15:40:09.1438777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10682
2022-09-27T15:40:10.1955825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:10.1956468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:10.2251300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:10.2251676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:10.2275837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:10.2276246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:10.2299280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:10.2299845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:10.2604007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:10.2604364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:10.2653374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:10.2653744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:10.5970678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:10.6071471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:10.6072559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:10.6073075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:10.6073593Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:10.6074127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:10.6078486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:10.6079795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:10.6080660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:10.6083403Z STAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6085355Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6086201Z STAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6092525Z STAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6093295Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6094085Z STAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6100468Z STAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6102567Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6103013Z 
2022-09-27T15:40:10.6110326Z STAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6111098Z STAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6111902Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6115954Z STAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6127532Z 
2022-09-27T15:40:10.6128065Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:10.6128803Z STAGE:2022-09-27 15:40:10 10681:10681 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6129472Z STAGE:2022-09-27 15:40:10 10680:10680 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:10.6129907Z STAGE:2022-09-27 15:40:10 10682:10682 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:11.0488294Z ok (2.147s)
2022-09-27T15:40:11.0488567Z 
2022-09-27T15:40:11.0489011Z ----------------------------------------------------------------------
2022-09-27T15:40:11.0489427Z Ran 1 test in 2.147s
2022-09-27T15:40:11.0489602Z 
2022-09-27T15:40:11.0489694Z OK
2022-09-27T15:40:11.0489822Z 
2022-09-27T15:40:11.0489968Z Generating XML reports...
2022-09-27T15:40:11.0535254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154008.xml
2022-09-27T15:40:12.3893971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:12.3894351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:12.3901750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:12.3902097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:12.7314043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:12.7323502Z 
2022-09-27T15:40:12.7323761Z Running tests...
2022-09-27T15:40:12.7324226Z ----------------------------------------------------------------------
2022-09-27T15:40:12.9682810Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10736
2022-09-27T15:40:12.9709650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10737
2022-09-27T15:40:12.9737501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10738
2022-09-27T15:40:14.0926263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:14.0927084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:14.0933360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:14.0934487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:14.1251745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:14.1252468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:14.1277264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:14.1277895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:14.1283245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:14.1283898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:14.1566871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:14.1567521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:14.4758146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:14.4830597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:14.4831331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:14.4831953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:14.4832474Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:14.4859609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:14.4937680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:14.4938701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:14.5870889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:14.8788106Z skip: CUDA is not available. (2.146s)
2022-09-27T15:40:14.8788288Z 
2022-09-27T15:40:14.8788745Z ----------------------------------------------------------------------
2022-09-27T15:40:14.8789187Z Ran 1 test in 2.146s
2022-09-27T15:40:14.8789403Z 
2022-09-27T15:40:14.8789545Z OK (skipped=1)
2022-09-27T15:40:14.8789730Z 
2022-09-27T15:40:14.8789840Z Generating XML reports...
2022-09-27T15:40:14.8834925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154012.xml
2022-09-27T15:40:16.1871277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:16.1871904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:16.1880129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:16.1880784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:16.5248409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:16.5259756Z 
2022-09-27T15:40:16.5260277Z Running tests...
2022-09-27T15:40:16.5260658Z ----------------------------------------------------------------------
2022-09-27T15:40:16.7620234Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10789
2022-09-27T15:40:16.7646279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10790
2022-09-27T15:40:16.7673496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10791
2022-09-27T15:40:17.8931080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:17.8931766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:17.9013420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:17.9013955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:17.9069389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:17.9069926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:17.9286407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:17.9287003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:17.9373277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:17.9373932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:17.9425437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:17.9426025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:18.2792023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:18.2792489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:18.2792906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:18.2793574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:18.2794158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:18.2794747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:18.2897764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:18.2901498Z STAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3802227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:18.3802832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:18.3807330Z STAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3807644Z 
2022-09-27T15:40:18.3812875Z STAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3813792Z STAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3814485Z STAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3822020Z STAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3823829Z STAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3824271Z 
2022-09-27T15:40:18.3832183Z STAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3832937Z STAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3833284Z 
2022-09-27T15:40:18.3838066Z STAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3840287Z STAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:18.3840805Z 
2022-09-27T15:40:18.3848177Z STAGE:2022-09-27 15:40:18 10791:10791 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3848985Z STAGE:2022-09-27 15:40:18 10790:10790 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.3849788Z STAGE:2022-09-27 15:40:18 10789:10789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:18.8723952Z ok (2.346s)
2022-09-27T15:40:18.8724154Z 
2022-09-27T15:40:18.8724679Z ----------------------------------------------------------------------
2022-09-27T15:40:18.8725088Z Ran 1 test in 2.346s
2022-09-27T15:40:18.8725211Z 
2022-09-27T15:40:18.8725274Z OK
2022-09-27T15:40:18.8725367Z 
2022-09-27T15:40:18.8725468Z Generating XML reports...
2022-09-27T15:40:18.8771156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154016.xml
2022-09-27T15:40:20.1806702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:20.1807373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:20.1815363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:20.1816018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:20.5191425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:20.5201451Z 
2022-09-27T15:40:20.5201892Z Running tests...
2022-09-27T15:40:20.5202292Z ----------------------------------------------------------------------
2022-09-27T15:40:20.7567665Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10845
2022-09-27T15:40:20.7594316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10846
2022-09-27T15:40:20.7621202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10847
2022-09-27T15:40:21.8542335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:21.8542715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:21.8811225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:21.8811902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:21.8860782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:21.8861439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:21.9141007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:21.9141703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:21.9266597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:21.9267216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:21.9611801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:21.9612465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:22.3039162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:22.3139918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:22.3140493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:22.3141128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:22.3141753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:22.3142263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:22.3247701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:22.3248083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:22.3251831Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.3252551Z STAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4150210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:22.4154657Z STAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4160517Z STAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4161536Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4162498Z STAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4170691Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4171139Z 
2022-09-27T15:40:22.4171553Z STAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4178557Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4179495Z STAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4179799Z 
2022-09-27T15:40:22.4184934Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4186364Z STAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:22.4186827Z 
2022-09-27T15:40:22.4193167Z STAGE:2022-09-27 15:40:22 10847:10847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4193703Z STAGE:2022-09-27 15:40:22 10846:10846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.4194160Z STAGE:2022-09-27 15:40:22 10845:10845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:22.7674638Z ok (2.247s)
2022-09-27T15:40:22.7674919Z 
2022-09-27T15:40:22.7675353Z ----------------------------------------------------------------------
2022-09-27T15:40:22.7675744Z Ran 1 test in 2.247s
2022-09-27T15:40:22.7675921Z 
2022-09-27T15:40:22.7676023Z OK
2022-09-27T15:40:22.7676176Z 
2022-09-27T15:40:22.7676329Z Generating XML reports...
2022-09-27T15:40:22.7723052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154020.xml
2022-09-27T15:40:24.1061236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:24.1061622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:24.1069866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:24.1070247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:24.4449973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:24.4459550Z 
2022-09-27T15:40:24.4459657Z Running tests...
2022-09-27T15:40:24.4460457Z ----------------------------------------------------------------------
2022-09-27T15:40:24.6793902Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10901
2022-09-27T15:40:24.6819891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10902
2022-09-27T15:40:24.6847254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10903
2022-09-27T15:40:25.7883065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:25.7883534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:25.7957776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:25.7958249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:25.8222694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:25.8223385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:25.8253714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:25.8254408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:25.8305414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:25.8305818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:25.8576555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:25.8577077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:26.1883987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:26.1984371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:26.1985262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:26.1986451Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:26.1987402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:26.1988209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:26.2092055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:26.2093021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:26.2097301Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.2098735Z STAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.2995846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:26.3001722Z STAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3007978Z STAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.3009233Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.3009768Z 
2022-09-27T15:40:26.3018016Z STAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3018390Z 
2022-09-27T15:40:26.3018651Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3026541Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.3027035Z 
2022-09-27T15:40:26.3027453Z STAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.3032618Z STAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3033309Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3033820Z STAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:40:26.3041741Z STAGE:2022-09-27 15:40:26 10903:10903 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:40:26 10901:10901 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.3042262Z 
2022-09-27T15:40:26.3042700Z STAGE:2022-09-27 15:40:26 10902:10902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:40:26.6898493Z ok (2.243s)
2022-09-27T15:40:26.6898716Z 
2022-09-27T15:40:26.6899186Z ----------------------------------------------------------------------
2022-09-27T15:40:26.6899623Z Ran 1 test in 2.244s
2022-09-27T15:40:26.6899830Z 
2022-09-27T15:40:26.6899974Z OK
2022-09-27T15:40:26.6900137Z 
2022-09-27T15:40:26.6900307Z Generating XML reports...
2022-09-27T15:40:26.6945584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154024.xml
2022-09-27T15:40:27.9962690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:27.9963073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:27.9970819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:27.9971255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:28.3365130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:28.3374824Z 
2022-09-27T15:40:28.3374937Z Running tests...
2022-09-27T15:40:28.3375429Z ----------------------------------------------------------------------
2022-09-27T15:40:28.5723350Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10957
2022-09-27T15:40:28.5749281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10958
2022-09-27T15:40:28.5776810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10959
2022-09-27T15:40:29.6263759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:29.6264266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:29.6577739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:29.6578374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:29.6579493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:29.6580147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:29.6607562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:29.6608382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:29.6929649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:29.6930237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:29.6961220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:29.6961852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:30.0315967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:30.0416608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:30.0417626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:30.0418050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:30.0418560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:30.0419067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:30.0424720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:30.0425552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:30.0426322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:30.3826887Z skip: CUDA is not available. (2.045s)
2022-09-27T15:40:30.3827199Z 
2022-09-27T15:40:30.3827663Z ----------------------------------------------------------------------
2022-09-27T15:40:30.3828155Z Ran 1 test in 2.045s
2022-09-27T15:40:30.3828370Z 
2022-09-27T15:40:30.3828490Z OK (skipped=1)
2022-09-27T15:40:30.3828637Z 
2022-09-27T15:40:30.3828775Z Generating XML reports...
2022-09-27T15:40:30.3873630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154028.xml
2022-09-27T15:40:31.6712872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:31.6713286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:31.6720986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:31.6721363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:32.0103363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:32.0113416Z 
2022-09-27T15:40:32.0113569Z Running tests...
2022-09-27T15:40:32.0114034Z ----------------------------------------------------------------------
2022-09-27T15:40:32.2457744Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11010
2022-09-27T15:40:32.2484172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11011
2022-09-27T15:40:32.2511047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11012
2022-09-27T15:40:33.3282476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:33.3283143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:33.3462353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:33.3462979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:33.3611617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:33.3612249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:33.3780788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:33.3781468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:33.3828206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:33.3828871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:33.4129088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:33.4129688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:33.7406936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:33.7507247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:33.7508248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:33.7508689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:33.7509257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:33.7510169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:33.7515250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:33.7515811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:33.7516245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:34.0560660Z skip: CUDA is not available. (2.044s)
2022-09-27T15:40:34.0560948Z 
2022-09-27T15:40:34.0561445Z ----------------------------------------------------------------------
2022-09-27T15:40:34.0561897Z Ran 1 test in 2.045s
2022-09-27T15:40:34.0562103Z 
2022-09-27T15:40:34.0562236Z OK (skipped=1)
2022-09-27T15:40:34.0562430Z 
2022-09-27T15:40:34.0562583Z Generating XML reports...
2022-09-27T15:40:34.0608303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154032.xml
2022-09-27T15:40:35.3389946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:35.3390303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:35.3398070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:35.3398588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:35.6763236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:35.6773423Z 
2022-09-27T15:40:35.6773637Z Running tests...
2022-09-27T15:40:35.9130859Z ----------------------------------------------------------------------
2022-09-27T15:40:35.9131699Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11063
2022-09-27T15:40:35.9157306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11064
2022-09-27T15:40:35.9184857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11065
2022-09-27T15:40:36.9481415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:36.9482321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:36.9801186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:36.9801810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:37.0049057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:37.0049617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:37.0366143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:37.0366847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:37.0367718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:37.0368376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:37.0685581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:37.0685988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:37.4217444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:40:37.4218090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:40:37.4218503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:40:37.4219116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:37.4219661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:37.4220172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:40:37.4224745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:40:37.4226259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:40:37.5229647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:40:37.8232755Z skip: CUDA is not available. (2.146s)
2022-09-27T15:40:37.8233096Z 
2022-09-27T15:40:37.8233769Z ----------------------------------------------------------------------
2022-09-27T15:40:37.8234190Z Ran 1 test in 2.146s
2022-09-27T15:40:37.8234311Z 
2022-09-27T15:40:37.8234392Z OK (skipped=1)
2022-09-27T15:40:37.8234500Z 
2022-09-27T15:40:37.8234600Z Generating XML reports...
2022-09-27T15:40:37.8280155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154035.xml
2022-09-27T15:40:39.1546122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:39.1546549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:39.1554307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:39.1554669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:39.4985212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:39.4995128Z 
2022-09-27T15:40:39.4995554Z Running tests...
2022-09-27T15:40:39.4995988Z ----------------------------------------------------------------------
2022-09-27T15:40:39.5012436Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:40:39.5012720Z 
2022-09-27T15:40:39.5012974Z ----------------------------------------------------------------------
2022-09-27T15:40:39.5013299Z Ran 1 test in 0.002s
2022-09-27T15:40:39.5013417Z 
2022-09-27T15:40:39.5013490Z OK (skipped=1)
2022-09-27T15:40:39.5013600Z 
2022-09-27T15:40:39.5013687Z Generating XML reports...
2022-09-27T15:40:39.5043306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154039.xml
2022-09-27T15:40:40.6930795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:40.6939276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:40.6940164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:40.6940794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:41.0318823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:41.0328454Z 
2022-09-27T15:40:41.0328866Z Running tests...
2022-09-27T15:40:41.0329446Z ----------------------------------------------------------------------
2022-09-27T15:40:41.0344861Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:40:41.0345332Z 
2022-09-27T15:40:41.0345720Z ----------------------------------------------------------------------
2022-09-27T15:40:41.0346120Z Ran 1 test in 0.002s
2022-09-27T15:40:41.0346301Z 
2022-09-27T15:40:41.0346418Z OK (skipped=1)
2022-09-27T15:40:41.0346595Z 
2022-09-27T15:40:41.0346736Z Generating XML reports...
2022-09-27T15:40:41.0376899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154041.xml
2022-09-27T15:40:42.2333056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:42.2333706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:42.2341861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:42.2342240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:42.5716919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:42.5726585Z 
2022-09-27T15:40:42.5726698Z Running tests...
2022-09-27T15:40:42.5727262Z ----------------------------------------------------------------------
2022-09-27T15:40:42.5742913Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-09-27T15:40:42.5743399Z 
2022-09-27T15:40:42.5744232Z ----------------------------------------------------------------------
2022-09-27T15:40:42.5744576Z Ran 1 test in 0.002s
2022-09-27T15:40:42.5744693Z 
2022-09-27T15:40:42.5744792Z OK (skipped=1)
2022-09-27T15:40:42.5744955Z 
2022-09-27T15:40:42.5745042Z Generating XML reports...
2022-09-27T15:40:42.5775677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154042.xml
2022-09-27T15:40:43.7703206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:43.7703572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:43.7711708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:43.7712124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:44.1090151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:44.1099634Z 
2022-09-27T15:40:44.1099732Z Running tests...
2022-09-27T15:40:44.1100759Z ----------------------------------------------------------------------
2022-09-27T15:40:44.1117074Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:40:44.1117685Z 
2022-09-27T15:40:44.1118061Z ----------------------------------------------------------------------
2022-09-27T15:40:44.1118315Z Ran 1 test in 0.002s
2022-09-27T15:40:44.1118432Z 
2022-09-27T15:40:44.1118505Z OK (skipped=1)
2022-09-27T15:40:44.1118613Z 
2022-09-27T15:40:44.1118687Z Generating XML reports...
2022-09-27T15:40:44.1147866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154044.xml
2022-09-27T15:40:45.2943561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:45.2944153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:45.2951973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:45.2952324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:45.6307683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:45.6316569Z 
2022-09-27T15:40:45.6316909Z Running tests...
2022-09-27T15:40:45.6317266Z ----------------------------------------------------------------------
2022-09-27T15:40:45.6332315Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:40:45.6332679Z 
2022-09-27T15:40:45.6333101Z ----------------------------------------------------------------------
2022-09-27T15:40:45.6333530Z Ran 1 test in 0.002s
2022-09-27T15:40:45.6333636Z 
2022-09-27T15:40:45.6333714Z OK (skipped=1)
2022-09-27T15:40:45.6333820Z 
2022-09-27T15:40:45.6333909Z Generating XML reports...
2022-09-27T15:40:45.6363765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154045.xml
2022-09-27T15:40:46.8421568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:46.8422241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:46.8430471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:46.8431118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:47.1810883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:47.1820602Z 
2022-09-27T15:40:47.1820926Z Running tests...
2022-09-27T15:40:47.1821525Z ----------------------------------------------------------------------
2022-09-27T15:40:47.1838073Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:40:47.1838493Z 
2022-09-27T15:40:47.1838863Z ----------------------------------------------------------------------
2022-09-27T15:40:47.1839265Z Ran 1 test in 0.002s
2022-09-27T15:40:47.1839465Z 
2022-09-27T15:40:47.1839587Z OK (skipped=1)
2022-09-27T15:40:47.1839761Z 
2022-09-27T15:40:47.1839905Z Generating XML reports...
2022-09-27T15:40:47.1869604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154047.xml
2022-09-27T15:40:48.3966157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:48.3966798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:48.3974996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:48.3975666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:48.7340929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:48.7351227Z 
2022-09-27T15:40:48.7351680Z Running tests...
2022-09-27T15:40:48.7352259Z ----------------------------------------------------------------------
2022-09-27T15:40:48.7367249Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:40:48.7367603Z 
2022-09-27T15:40:48.7367970Z ----------------------------------------------------------------------
2022-09-27T15:40:48.7368406Z Ran 1 test in 0.002s
2022-09-27T15:40:48.7368612Z 
2022-09-27T15:40:48.7368746Z OK (skipped=1)
2022-09-27T15:40:48.7368940Z 
2022-09-27T15:40:48.7369049Z Generating XML reports...
2022-09-27T15:40:48.7398644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154048.xml
2022-09-27T15:40:49.9253305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:49.9253697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:49.9261694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:49.9262239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:50.2643646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:50.2653379Z 
2022-09-27T15:40:50.2653517Z Running tests...
2022-09-27T15:40:50.2653971Z ----------------------------------------------------------------------
2022-09-27T15:40:50.2670779Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:40:50.2671112Z 
2022-09-27T15:40:50.2671353Z ----------------------------------------------------------------------
2022-09-27T15:40:50.2671600Z Ran 1 test in 0.002s
2022-09-27T15:40:50.2671714Z 
2022-09-27T15:40:50.2671814Z OK (skipped=1)
2022-09-27T15:40:50.2671969Z 
2022-09-27T15:40:50.2672059Z Generating XML reports...
2022-09-27T15:40:50.2702088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154050.xml
2022-09-27T15:40:51.4783211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:51.4783817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:51.4791916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:51.4792550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:51.8180940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:51.8191645Z 
2022-09-27T15:40:51.8191857Z Running tests...
2022-09-27T15:40:51.8192417Z ----------------------------------------------------------------------
2022-09-27T15:40:51.8208122Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:40:51.8208546Z 
2022-09-27T15:40:51.8208933Z ----------------------------------------------------------------------
2022-09-27T15:40:51.8209237Z Ran 1 test in 0.002s
2022-09-27T15:40:51.8209395Z 
2022-09-27T15:40:51.8209498Z OK (skipped=1)
2022-09-27T15:40:51.8209614Z 
2022-09-27T15:40:51.8209706Z Generating XML reports...
2022-09-27T15:40:51.8239292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154051.xml
2022-09-27T15:40:53.0114861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:53.0115222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:53.0123939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:53.0124330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:53.3572913Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:53.3583257Z 
2022-09-27T15:40:53.3583540Z Running tests...
2022-09-27T15:40:53.3584365Z ----------------------------------------------------------------------
2022-09-27T15:40:53.3600158Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:40:53.3600604Z 
2022-09-27T15:40:53.3600982Z ----------------------------------------------------------------------
2022-09-27T15:40:53.3601381Z Ran 1 test in 0.002s
2022-09-27T15:40:53.3601569Z 
2022-09-27T15:40:53.3601692Z OK (skipped=1)
2022-09-27T15:40:53.3601862Z 
2022-09-27T15:40:53.3602011Z Generating XML reports...
2022-09-27T15:40:53.3631906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154053.xml
2022-09-27T15:40:54.5503776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:54.5504713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:54.5513018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:54.5513669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:54.8874865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:54.8884629Z 
2022-09-27T15:40:54.8884918Z Running tests...
2022-09-27T15:40:54.8885528Z ----------------------------------------------------------------------
2022-09-27T15:40:54.8902229Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:40:54.8902735Z 
2022-09-27T15:40:54.8903132Z ----------------------------------------------------------------------
2022-09-27T15:40:54.8903614Z Ran 1 test in 0.002s
2022-09-27T15:40:54.8903824Z 
2022-09-27T15:40:54.8904100Z OK (skipped=1)
2022-09-27T15:40:54.8904216Z 
2022-09-27T15:40:54.8904290Z Generating XML reports...
2022-09-27T15:40:54.8933197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154054.xml
2022-09-27T15:40:56.0752121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:56.0752496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:56.0759987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:56.0760382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:56.4143813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:56.4153222Z 
2022-09-27T15:40:56.4153331Z Running tests...
2022-09-27T15:40:56.4153938Z ----------------------------------------------------------------------
2022-09-27T15:40:56.4170433Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:40:56.4170792Z 
2022-09-27T15:40:56.4171150Z ----------------------------------------------------------------------
2022-09-27T15:40:56.4171393Z Ran 1 test in 0.002s
2022-09-27T15:40:56.4171510Z 
2022-09-27T15:40:56.4171591Z OK (skipped=1)
2022-09-27T15:40:56.4171700Z 
2022-09-27T15:40:56.4171787Z Generating XML reports...
2022-09-27T15:40:56.4201538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154056.xml
2022-09-27T15:40:57.6023985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:57.6024407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:57.6031992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:57.6032341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:57.9392109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:57.9401827Z 
2022-09-27T15:40:57.9401911Z Running tests...
2022-09-27T15:40:57.9402490Z ----------------------------------------------------------------------
2022-09-27T15:40:57.9418101Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:40:57.9418622Z 
2022-09-27T15:40:57.9419025Z ----------------------------------------------------------------------
2022-09-27T15:40:57.9419433Z Ran 1 test in 0.002s
2022-09-27T15:40:57.9419550Z 
2022-09-27T15:40:57.9419627Z OK (skipped=1)
2022-09-27T15:40:57.9419737Z 
2022-09-27T15:40:57.9419821Z Generating XML reports...
2022-09-27T15:40:57.9448650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154057.xml
2022-09-27T15:40:59.1322507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:40:59.1322895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:40:59.1330170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:40:59.1330539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:40:59.4715389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:40:59.4724900Z 
2022-09-27T15:40:59.4725036Z Running tests...
2022-09-27T15:40:59.4725848Z ----------------------------------------------------------------------
2022-09-27T15:40:59.4743711Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:40:59.4744168Z 
2022-09-27T15:40:59.4744527Z ----------------------------------------------------------------------
2022-09-27T15:40:59.4744871Z Ran 1 test in 0.002s
2022-09-27T15:40:59.4744990Z 
2022-09-27T15:40:59.4745065Z OK (skipped=1)
2022-09-27T15:40:59.4745264Z 
2022-09-27T15:40:59.4745384Z Generating XML reports...
2022-09-27T15:40:59.4774036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154059.xml
2022-09-27T15:41:00.6574276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:00.6574647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:00.6582795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:00.6583171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:00.9943642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:00.9954421Z 
2022-09-27T15:41:00.9954860Z Running tests...
2022-09-27T15:41:00.9955260Z ----------------------------------------------------------------------
2022-09-27T15:41:00.9970950Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:41:00.9971427Z 
2022-09-27T15:41:00.9971831Z ----------------------------------------------------------------------
2022-09-27T15:41:00.9972096Z Ran 1 test in 0.002s
2022-09-27T15:41:00.9972214Z 
2022-09-27T15:41:00.9972290Z OK (skipped=1)
2022-09-27T15:41:00.9972400Z 
2022-09-27T15:41:00.9972651Z Generating XML reports...
2022-09-27T15:41:01.0001315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154100.xml
2022-09-27T15:41:02.2272736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:02.2273190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:02.2280575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:02.2280934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:02.5665098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:02.5674496Z 
2022-09-27T15:41:02.5674584Z Running tests...
2022-09-27T15:41:02.5675509Z ----------------------------------------------------------------------
2022-09-27T15:41:02.5692106Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:41:02.5692554Z 
2022-09-27T15:41:02.5692925Z ----------------------------------------------------------------------
2022-09-27T15:41:02.5693343Z Ran 1 test in 0.002s
2022-09-27T15:41:02.5693542Z 
2022-09-27T15:41:02.5693668Z OK (skipped=1)
2022-09-27T15:41:02.5693864Z 
2022-09-27T15:41:02.5694013Z Generating XML reports...
2022-09-27T15:41:02.5723662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154102.xml
2022-09-27T15:41:03.7672313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:03.7672954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:03.7681309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:03.7681961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:04.1070327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:04.1080446Z 
2022-09-27T15:41:04.1080952Z Running tests...
2022-09-27T15:41:04.1081352Z ----------------------------------------------------------------------
2022-09-27T15:41:04.1096533Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:41:04.1096810Z 
2022-09-27T15:41:04.1097128Z ----------------------------------------------------------------------
2022-09-27T15:41:04.1097379Z Ran 1 test in 0.002s
2022-09-27T15:41:04.1097483Z 
2022-09-27T15:41:04.1097557Z OK (skipped=1)
2022-09-27T15:41:04.1097689Z 
2022-09-27T15:41:04.1097814Z Generating XML reports...
2022-09-27T15:41:04.1126958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154104.xml
2022-09-27T15:41:05.3248669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:05.3249087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:05.3257145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:05.3257506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:05.6633495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:05.6642624Z 
2022-09-27T15:41:05.6642749Z Running tests...
2022-09-27T15:41:05.6643193Z ----------------------------------------------------------------------
2022-09-27T15:41:05.6659229Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:41:05.6659611Z 
2022-09-27T15:41:05.6660338Z ----------------------------------------------------------------------
2022-09-27T15:41:05.6660590Z Ran 1 test in 0.002s
2022-09-27T15:41:05.6660704Z 
2022-09-27T15:41:05.6660779Z OK (skipped=1)
2022-09-27T15:41:05.6660959Z 
2022-09-27T15:41:05.6661042Z Generating XML reports...
2022-09-27T15:41:05.6690808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154105.xml
2022-09-27T15:41:06.8720783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:06.8721153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:06.8729335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:06.8729722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:07.2097040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:07.2106572Z 
2022-09-27T15:41:07.2106812Z Running tests...
2022-09-27T15:41:07.2107153Z ----------------------------------------------------------------------
2022-09-27T15:41:07.2123913Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:41:07.2124343Z 
2022-09-27T15:41:07.2124722Z ----------------------------------------------------------------------
2022-09-27T15:41:07.2125158Z Ran 1 test in 0.002s
2022-09-27T15:41:07.2125307Z 
2022-09-27T15:41:07.2125382Z OK (skipped=1)
2022-09-27T15:41:07.2125494Z 
2022-09-27T15:41:07.2125578Z Generating XML reports...
2022-09-27T15:41:07.2155965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154107.xml
2022-09-27T15:41:08.4326350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:08.4327000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:08.4334925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:08.4335319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:08.7715905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:08.7725168Z 
2022-09-27T15:41:08.7725359Z Running tests...
2022-09-27T15:41:08.7725802Z ----------------------------------------------------------------------
2022-09-27T15:41:08.7742862Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:41:08.7743284Z 
2022-09-27T15:41:08.7743751Z ----------------------------------------------------------------------
2022-09-27T15:41:08.7744254Z Ran 1 test in 0.002s
2022-09-27T15:41:08.7744373Z 
2022-09-27T15:41:08.7744460Z OK (skipped=1)
2022-09-27T15:41:08.7744569Z 
2022-09-27T15:41:08.7744642Z Generating XML reports...
2022-09-27T15:41:08.7774021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154108.xml
2022-09-27T15:41:09.9571214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:09.9571650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:09.9579526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:09.9579887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:10.2954951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:10.2964324Z 
2022-09-27T15:41:10.2964488Z Running tests...
2022-09-27T15:41:10.2965209Z ----------------------------------------------------------------------
2022-09-27T15:41:10.2981244Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:41:10.2981828Z 
2022-09-27T15:41:10.2982678Z ----------------------------------------------------------------------
2022-09-27T15:41:10.2983113Z Ran 1 test in 0.002s
2022-09-27T15:41:10.2983220Z 
2022-09-27T15:41:10.2983299Z OK (skipped=1)
2022-09-27T15:41:10.2983414Z 
2022-09-27T15:41:10.2983501Z Generating XML reports...
2022-09-27T15:41:10.3012672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154110.xml
2022-09-27T15:41:11.5170685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:11.5171073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:11.5179109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:11.5179472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:11.8553112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:11.8562947Z 
2022-09-27T15:41:11.8563507Z Running tests...
2022-09-27T15:41:11.8563949Z ----------------------------------------------------------------------
2022-09-27T15:41:11.8579731Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:41:11.8580596Z 
2022-09-27T15:41:11.8580889Z ----------------------------------------------------------------------
2022-09-27T15:41:11.8581329Z Ran 1 test in 0.002s
2022-09-27T15:41:11.8581519Z 
2022-09-27T15:41:11.8581655Z OK (skipped=1)
2022-09-27T15:41:11.8581784Z 
2022-09-27T15:41:11.8581873Z Generating XML reports...
2022-09-27T15:41:11.8611007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154111.xml
2022-09-27T15:41:13.0750005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:13.0750473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:13.0758218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:13.0758579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:13.4145544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:13.4155264Z 
2022-09-27T15:41:13.4155359Z Running tests...
2022-09-27T15:41:13.4155817Z ----------------------------------------------------------------------
2022-09-27T15:41:13.4171721Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:41:13.4172207Z 
2022-09-27T15:41:13.4172507Z ----------------------------------------------------------------------
2022-09-27T15:41:13.4172770Z Ran 1 test in 0.002s
2022-09-27T15:41:13.4172886Z 
2022-09-27T15:41:13.4172960Z OK (skipped=1)
2022-09-27T15:41:13.4173067Z 
2022-09-27T15:41:13.4173155Z Generating XML reports...
2022-09-27T15:41:13.4202614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154113.xml
2022-09-27T15:41:14.6140021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:14.6140397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:14.6148077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:14.6148441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:14.9522986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:14.9532218Z 
2022-09-27T15:41:14.9532327Z Running tests...
2022-09-27T15:41:14.9532927Z ----------------------------------------------------------------------
2022-09-27T15:41:14.9549833Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:41:14.9550113Z 
2022-09-27T15:41:14.9550491Z ----------------------------------------------------------------------
2022-09-27T15:41:14.9550921Z Ran 1 test in 0.002s
2022-09-27T15:41:14.9551071Z 
2022-09-27T15:41:14.9551148Z OK (skipped=1)
2022-09-27T15:41:14.9551258Z 
2022-09-27T15:41:14.9551344Z Generating XML reports...
2022-09-27T15:41:14.9581618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154114.xml
2022-09-27T15:41:16.1670014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:16.1670689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:16.1679477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:16.1680108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:16.5044475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:16.5054659Z 
2022-09-27T15:41:16.5055252Z Running tests...
2022-09-27T15:41:16.5055814Z ----------------------------------------------------------------------
2022-09-27T15:41:16.7428584Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11356
2022-09-27T15:41:16.7454979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11357
2022-09-27T15:41:16.7481611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11358
2022-09-27T15:41:17.8436451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:17.8437084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:17.8659854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:17.8660550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:17.8770592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:17.8771058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:17.8959436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:17.8960135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:17.9005714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:17.9006102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:17.9282212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:17.9282730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:18.2461645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:18.2558001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:18.2558480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:18.2559308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:18.2559847Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:18.2562593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:18.2666234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:18.2667137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:18.3574406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:18.6530938Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:41:18.6531229Z 
2022-09-27T15:41:18.6531706Z ----------------------------------------------------------------------
2022-09-27T15:41:18.6532097Z Ran 1 test in 2.148s
2022-09-27T15:41:18.6532270Z 
2022-09-27T15:41:18.6532377Z OK (skipped=1)
2022-09-27T15:41:18.6532554Z 
2022-09-27T15:41:18.6532697Z Generating XML reports...
2022-09-27T15:41:18.6578164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154116.xml
2022-09-27T15:41:19.9486971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:19.9487345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:19.9494080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:19.9494510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:20.2857820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:20.2867969Z 
2022-09-27T15:41:20.2868523Z Running tests...
2022-09-27T15:41:20.2868946Z ----------------------------------------------------------------------
2022-09-27T15:41:20.2884633Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:41:20.2885114Z 
2022-09-27T15:41:20.2885360Z ----------------------------------------------------------------------
2022-09-27T15:41:20.2885606Z Ran 1 test in 0.002s
2022-09-27T15:41:20.2885709Z 
2022-09-27T15:41:20.2885780Z OK (skipped=1)
2022-09-27T15:41:20.2885888Z 
2022-09-27T15:41:20.2885973Z Generating XML reports...
2022-09-27T15:41:20.2914768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154120.xml
2022-09-27T15:41:21.4909986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:21.4910646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:21.4919244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:21.4919897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:21.8440218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:21.8450364Z 
2022-09-27T15:41:21.8450785Z Running tests...
2022-09-27T15:41:21.8451162Z ----------------------------------------------------------------------
2022-09-27T15:41:21.8467310Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:41:21.8467716Z 
2022-09-27T15:41:21.8468063Z ----------------------------------------------------------------------
2022-09-27T15:41:21.8468508Z Ran 1 test in 0.002s
2022-09-27T15:41:21.8468696Z 
2022-09-27T15:41:21.8468779Z OK (skipped=1)
2022-09-27T15:41:21.8468890Z 
2022-09-27T15:41:21.8468975Z Generating XML reports...
2022-09-27T15:41:21.8498817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154121.xml
2022-09-27T15:41:23.0514273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:23.0514863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:23.0522540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:23.0523039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:23.3971268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:23.3980792Z 
2022-09-27T15:41:23.3980890Z Running tests...
2022-09-27T15:41:23.3981616Z ----------------------------------------------------------------------
2022-09-27T15:41:23.6404950Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11429
2022-09-27T15:41:23.6431887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11430
2022-09-27T15:41:23.6458444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11431
2022-09-27T15:41:24.7252929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:24.7253577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:24.7555977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:24.7556680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:24.7567557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:24.7568209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:24.7852978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:24.7853365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:24.7881679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:24.7882182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:24.8177012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:24.8177669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:25.1436803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:25.1537711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:25.1538119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:25.1538956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:25.1539854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:25.1540636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:25.1644017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:25.2547726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:25.2548190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:26.8523689Z ok (3.454s)
2022-09-27T15:41:26.8523957Z 
2022-09-27T15:41:26.8524500Z ----------------------------------------------------------------------
2022-09-27T15:41:26.8525238Z Ran 1 test in 3.454s
2022-09-27T15:41:26.8525466Z 
2022-09-27T15:41:26.8525580Z OK
2022-09-27T15:41:26.8525755Z 
2022-09-27T15:41:26.8525913Z Generating XML reports...
2022-09-27T15:41:26.8572513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154123.xml
2022-09-27T15:41:28.1599481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:28.1599861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:28.1607678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:28.1608029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:28.4990520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:28.4999795Z 
2022-09-27T15:41:28.5000028Z Running tests...
2022-09-27T15:41:28.5000491Z ----------------------------------------------------------------------
2022-09-27T15:41:28.7340677Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11482
2022-09-27T15:41:28.7367372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11483
2022-09-27T15:41:28.7394964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11484
2022-09-27T15:41:29.8222814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:29.8223543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:29.8539980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:29.8540606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:29.8611521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:29.8612143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:29.8653724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:29.8654092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:29.8960118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:29.8960732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:29.8998678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:29.8999314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:30.2330942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:30.2432116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:30.2432530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:30.2433163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:30.2433685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:30.2434210Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:30.2538648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:30.3445183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:30.3445989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:30.6547310Z skip: CUDA is not available. (2.154s)
2022-09-27T15:41:30.6547559Z 
2022-09-27T15:41:30.6548247Z ----------------------------------------------------------------------
2022-09-27T15:41:30.6548666Z Ran 1 test in 2.155s
2022-09-27T15:41:30.6548841Z 
2022-09-27T15:41:30.6548958Z OK (skipped=1)
2022-09-27T15:41:30.6549124Z 
2022-09-27T15:41:30.6549234Z Generating XML reports...
2022-09-27T15:41:30.6594121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154128.xml
2022-09-27T15:41:31.9935873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:31.9936268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:31.9943798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:31.9944270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:32.3374336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:32.3384622Z 
2022-09-27T15:41:32.3384889Z Running tests...
2022-09-27T15:41:32.3385354Z ----------------------------------------------------------------------
2022-09-27T15:41:32.5745265Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11535
2022-09-27T15:41:32.5772722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11536
2022-09-27T15:41:32.5799555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11537
2022-09-27T15:41:33.6222035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:33.6222637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:33.6326262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:33.6326803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:33.6558566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:33.6559192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:33.6667872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:33.6668519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:33.6699517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:33.6699883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:33.7028019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:33.7028427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:34.0470824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:34.0471325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:34.0472046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:34.0472517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:34.0473065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:34.0473808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:34.0577720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:34.0578274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:34.1480790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:34.1597919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:41:34.1691811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:41:34.1692204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:41:34.1692827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:34.1693547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:34.1699955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:35.7866214Z ok (3.448s)
2022-09-27T15:41:35.7866444Z 
2022-09-27T15:41:35.7866984Z ----------------------------------------------------------------------
2022-09-27T15:41:35.7867294Z Ran 1 test in 3.448s
2022-09-27T15:41:35.7867409Z 
2022-09-27T15:41:35.7867472Z OK
2022-09-27T15:41:35.7867564Z 
2022-09-27T15:41:35.7867642Z Generating XML reports...
2022-09-27T15:41:35.7913573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154132.xml
2022-09-27T15:41:37.0910918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:37.0911560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:37.0919639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:37.0920311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:37.4416761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:37.4427528Z 
2022-09-27T15:41:37.4427882Z Running tests...
2022-09-27T15:41:37.4428588Z ----------------------------------------------------------------------
2022-09-27T15:41:37.6781004Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11597
2022-09-27T15:41:37.6808661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11598
2022-09-27T15:41:37.6835678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11599
2022-09-27T15:41:38.8032626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:38.8033325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:38.8170780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:38.8171532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:38.8239029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:38.8239539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:38.8372213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:38.8372875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:38.8529977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:38.8530899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:38.8591771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:38.8592685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:39.1967450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:39.1968161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:39.1968784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:39.1969732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:39.1970474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:39.1971045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:39.1974836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:39.1975688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:39.1979400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:39.4881804Z skip: CUDA is not available. (2.045s)
2022-09-27T15:41:39.4882070Z 
2022-09-27T15:41:39.4882525Z ----------------------------------------------------------------------
2022-09-27T15:41:39.4882950Z Ran 1 test in 2.045s
2022-09-27T15:41:39.4883110Z 
2022-09-27T15:41:39.4883219Z OK (skipped=1)
2022-09-27T15:41:39.4883389Z 
2022-09-27T15:41:39.4883537Z Generating XML reports...
2022-09-27T15:41:39.4928126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154137.xml
2022-09-27T15:41:40.7714999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:40.7715677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:40.7724079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:40.7724709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:41.1113462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:41.1123470Z 
2022-09-27T15:41:41.1123743Z Running tests...
2022-09-27T15:41:41.1124361Z ----------------------------------------------------------------------
2022-09-27T15:41:41.3497284Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11650
2022-09-27T15:41:41.3524300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11651
2022-09-27T15:41:41.3551792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11652
2022-09-27T15:41:42.4140233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:42.4140667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:42.4195436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:42.4195783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:42.4551383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:42.4551741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:42.4603398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:42.4603993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:42.4662555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:42.4663015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:42.4978915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:42.4979282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:42.8332182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:42.8433350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:42.8434047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:42.8434732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:42.8435262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:42.8435791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:42.8540086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:42.9443588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:42.9444168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:42.9445046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:41:42.9551043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:41:42.9551603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:41:42.9552224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:42.9552975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:42.9648919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:44.1612534Z ok (3.049s)
2022-09-27T15:41:44.1612757Z 
2022-09-27T15:41:44.1613299Z ----------------------------------------------------------------------
2022-09-27T15:41:44.1613700Z Ran 1 test in 3.049s
2022-09-27T15:41:44.1613827Z 
2022-09-27T15:41:44.1613892Z OK
2022-09-27T15:41:44.1613993Z 
2022-09-27T15:41:44.1614086Z Generating XML reports...
2022-09-27T15:41:44.1661678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154141.xml
2022-09-27T15:41:45.4867186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:45.4867636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:45.4875413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:45.4875773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:45.8280793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:45.8290680Z 
2022-09-27T15:41:45.8290848Z Running tests...
2022-09-27T15:41:45.8291193Z ----------------------------------------------------------------------
2022-09-27T15:41:46.0631328Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11709
2022-09-27T15:41:46.0657530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11710
2022-09-27T15:41:46.0684674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11711
2022-09-27T15:41:47.1687587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:47.1688246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:47.1861717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:47.1862257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:47.2004099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:47.2004788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:47.2020302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:47.2020993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:47.2219801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:47.2220331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:47.2341377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:47.2342020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:47.5693947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:47.5694466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:47.5694862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:47.5695479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:47.5696003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:47.5696528Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:47.5701464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:47.5703055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:47.5703694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:47.8732448Z skip: CUDA is not available. (2.044s)
2022-09-27T15:41:47.8732757Z 
2022-09-27T15:41:47.8733255Z ----------------------------------------------------------------------
2022-09-27T15:41:47.8733705Z Ran 1 test in 2.044s
2022-09-27T15:41:47.8733904Z 
2022-09-27T15:41:47.8734032Z OK (skipped=1)
2022-09-27T15:41:47.8734243Z 
2022-09-27T15:41:47.8734395Z Generating XML reports...
2022-09-27T15:41:47.8780783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154145.xml
2022-09-27T15:41:49.1620994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:49.1621375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:49.1629409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:49.1629761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:49.5007736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:49.5017324Z 
2022-09-27T15:41:49.5017449Z Running tests...
2022-09-27T15:41:49.5017807Z ----------------------------------------------------------------------
2022-09-27T15:41:49.7377707Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11762
2022-09-27T15:41:49.7403488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11763
2022-09-27T15:41:49.7430865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11764
2022-09-27T15:41:50.8373743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:50.8374127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:50.8697834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:50.8698321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:50.8735997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:50.8736630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:50.8919827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:50.8920496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:50.9080386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:50.9080952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:50.9252749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:50.9253375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:51.2430700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:51.2532017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:51.2532485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:51.2533336Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:51.2534200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:51.2534979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:51.2638298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:51.3542221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:51.3543817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:51.3853035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:41:51.3853671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:41:51.3854190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:41:51.3854919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:51.3855716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:51.3856510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:52.7494559Z ok (3.247s)
2022-09-27T15:41:52.7494737Z 
2022-09-27T15:41:52.7495082Z ----------------------------------------------------------------------
2022-09-27T15:41:52.7495331Z Ran 1 test in 3.248s
2022-09-27T15:41:52.7495520Z 
2022-09-27T15:41:52.7495569Z OK
2022-09-27T15:41:52.7495659Z 
2022-09-27T15:41:52.7495751Z Generating XML reports...
2022-09-27T15:41:52.7541996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154149.xml
2022-09-27T15:41:54.0655112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:54.0655500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:54.0663236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:54.0663695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:54.4050947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:54.4060430Z 
2022-09-27T15:41:54.4060576Z Running tests...
2022-09-27T15:41:54.4061052Z ----------------------------------------------------------------------
2022-09-27T15:41:54.4080564Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-09-27T15:41:54.4080928Z 
2022-09-27T15:41:54.4081155Z ----------------------------------------------------------------------
2022-09-27T15:41:54.4081400Z Ran 1 test in 0.002s
2022-09-27T15:41:54.4081514Z 
2022-09-27T15:41:54.4081575Z OK (skipped=1)
2022-09-27T15:41:54.4081687Z 
2022-09-27T15:41:54.4081773Z Generating XML reports...
2022-09-27T15:41:54.4111618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154154.xml
2022-09-27T15:41:55.5922805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:55.5923175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:55.5931200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:55.5931565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:55.9311510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:41:55.9321065Z 
2022-09-27T15:41:55.9321201Z Running tests...
2022-09-27T15:41:55.9321798Z ----------------------------------------------------------------------
2022-09-27T15:41:56.1670539Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11834
2022-09-27T15:41:56.1696176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11835
2022-09-27T15:41:56.1724530Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11836
2022-09-27T15:41:57.2677917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:57.2678557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:57.2792673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:57.2793282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:57.3016700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:57.3017361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:57.3111758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:41:57.3112743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:41:57.3137063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:57.3137965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:57.3432895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:41:57.3433534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:41:57.6920723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:41:57.6921445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:41:57.6921908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:41:57.6922960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:57.6923578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:57.6924117Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:41:57.7026661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:41:57.7930363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:41:57.7931367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:41:57.7932098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:41:57.8036916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:41:57.8037620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:41:57.8038490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:57.8039375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:41:57.8136528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:03.1840697Z ok (7.251s)
2022-09-27T15:42:03.1840949Z 
2022-09-27T15:42:03.1841448Z ----------------------------------------------------------------------
2022-09-27T15:42:03.1841758Z Ran 1 test in 7.252s
2022-09-27T15:42:03.1841876Z 
2022-09-27T15:42:03.1841940Z OK
2022-09-27T15:42:03.1842082Z 
2022-09-27T15:42:03.1842186Z Generating XML reports...
2022-09-27T15:42:03.1887845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154155.xml
2022-09-27T15:42:04.4812045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:04.4812475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:04.4820206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:04.4820594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:04.8189648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:04.8199030Z 
2022-09-27T15:42:04.8199411Z Running tests...
2022-09-27T15:42:04.8199837Z ----------------------------------------------------------------------
2022-09-27T15:42:05.0558479Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11893
2022-09-27T15:42:05.0584222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11894
2022-09-27T15:42:05.0611427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11895
2022-09-27T15:42:06.1542601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:06.1543261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:06.1579046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:06.1579693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:06.1851151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:06.1851837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:06.1892326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:06.1892956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:06.1934347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:06.1934984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:06.2206772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:06.2207454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:06.5553497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:06.5654906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:06.5656460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:06.5657146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:06.5657866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:06.5658393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:06.5662617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:06.5663424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:06.5664645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:06.8659396Z ok (2.046s)
2022-09-27T15:42:06.8659639Z 
2022-09-27T15:42:06.8660114Z ----------------------------------------------------------------------
2022-09-27T15:42:06.8660592Z Ran 1 test in 2.046s
2022-09-27T15:42:06.8660741Z 
2022-09-27T15:42:06.8660829Z OK
2022-09-27T15:42:06.8660954Z 
2022-09-27T15:42:06.8662974Z Generating XML reports...
2022-09-27T15:42:06.8706497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154204.xml
2022-09-27T15:42:08.1530150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:08.1530570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:08.1538080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:08.1538628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:08.4918518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:08.4927630Z 
2022-09-27T15:42:08.4927769Z Running tests...
2022-09-27T15:42:08.4928693Z ----------------------------------------------------------------------
2022-09-27T15:42:08.7272429Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11946
2022-09-27T15:42:08.7298272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11947
2022-09-27T15:42:08.7325635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11948
2022-09-27T15:42:09.8220517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:09.8233168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:09.8529299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:09.8529897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:09.8769901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:09.8770560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:09.9004262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:09.9004883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:09.9098504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:09.9098944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:09.9332733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:09.9333152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:10.2500540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:10.2601538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:10.2602000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:10.2602621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:10.2603151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:10.2603675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:10.2708072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:10.3613036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:10.3613556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:10.7375407Z ok (2.244s)
2022-09-27T15:42:10.7375569Z 
2022-09-27T15:42:10.7376038Z ----------------------------------------------------------------------
2022-09-27T15:42:10.7376515Z Ran 1 test in 2.245s
2022-09-27T15:42:10.7376727Z 
2022-09-27T15:42:10.7376817Z OK
2022-09-27T15:42:10.7376928Z 
2022-09-27T15:42:10.7377021Z Generating XML reports...
2022-09-27T15:42:10.7422760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154208.xml
2022-09-27T15:42:12.0626961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:12.0627339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:12.0635257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:12.0635865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:12.4001486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:12.4010716Z 
2022-09-27T15:42:12.4010849Z Running tests...
2022-09-27T15:42:12.4011432Z ----------------------------------------------------------------------
2022-09-27T15:42:12.4033213Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:12.4033478Z 
2022-09-27T15:42:12.4033802Z ----------------------------------------------------------------------
2022-09-27T15:42:12.4034052Z Ran 1 test in 0.002s
2022-09-27T15:42:12.4034167Z 
2022-09-27T15:42:12.4034228Z OK (skipped=1)
2022-09-27T15:42:12.4034369Z 
2022-09-27T15:42:12.4034486Z Generating XML reports...
2022-09-27T15:42:12.4063133Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154212.xml
2022-09-27T15:42:13.6187013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:13.6187390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:13.6194863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:13.6195226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:13.9566501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:13.9576003Z 
2022-09-27T15:42:13.9576228Z Running tests...
2022-09-27T15:42:13.9576564Z ----------------------------------------------------------------------
2022-09-27T15:42:13.9603091Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:13.9603465Z 
2022-09-27T15:42:13.9603899Z ----------------------------------------------------------------------
2022-09-27T15:42:13.9604302Z Ran 1 test in 0.003s
2022-09-27T15:42:13.9604491Z 
2022-09-27T15:42:13.9604613Z OK (skipped=1)
2022-09-27T15:42:13.9604810Z 
2022-09-27T15:42:13.9604944Z Generating XML reports...
2022-09-27T15:42:13.9635273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154213.xml
2022-09-27T15:42:15.1450084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:15.1450595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:15.1472992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:15.1473386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:15.4858438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:15.4868215Z 
2022-09-27T15:42:15.4868329Z Running tests...
2022-09-27T15:42:15.4868871Z ----------------------------------------------------------------------
2022-09-27T15:42:15.4891649Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:15.4891964Z 
2022-09-27T15:42:15.4892282Z ----------------------------------------------------------------------
2022-09-27T15:42:15.4892607Z Ran 1 test in 0.002s
2022-09-27T15:42:15.4892728Z 
2022-09-27T15:42:15.4892803Z OK (skipped=1)
2022-09-27T15:42:15.4892899Z 
2022-09-27T15:42:15.4892986Z Generating XML reports...
2022-09-27T15:42:15.4922431Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154215.xml
2022-09-27T15:42:16.7016118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:16.7016789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:16.7025261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:16.7025852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:17.0411262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:17.0421107Z 
2022-09-27T15:42:17.0421320Z Running tests...
2022-09-27T15:42:17.0421662Z ----------------------------------------------------------------------
2022-09-27T15:42:17.0440444Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:17.0440828Z 
2022-09-27T15:42:17.0441150Z ----------------------------------------------------------------------
2022-09-27T15:42:17.0441488Z Ran 1 test in 0.002s
2022-09-27T15:42:17.0441611Z 
2022-09-27T15:42:17.0441694Z OK (skipped=1)
2022-09-27T15:42:17.0441806Z 
2022-09-27T15:42:17.0441917Z Generating XML reports...
2022-09-27T15:42:17.0471799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154217.xml
2022-09-27T15:42:18.2558111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:18.2558533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:18.2565900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:18.2566267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:18.5967081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:18.5977012Z 
2022-09-27T15:42:18.5977149Z Running tests...
2022-09-27T15:42:18.5977739Z ----------------------------------------------------------------------
2022-09-27T15:42:18.5995232Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:18.5995662Z 
2022-09-27T15:42:18.5996109Z ----------------------------------------------------------------------
2022-09-27T15:42:18.5996450Z Ran 1 test in 0.002s
2022-09-27T15:42:18.5996552Z 
2022-09-27T15:42:18.5996625Z OK (skipped=1)
2022-09-27T15:42:18.5996735Z 
2022-09-27T15:42:18.5996822Z Generating XML reports...
2022-09-27T15:42:18.6025606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154218.xml
2022-09-27T15:42:19.8130480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:19.8131157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:19.8139562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:19.8140225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:20.1512580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:20.1522576Z 
2022-09-27T15:42:20.1523061Z Running tests...
2022-09-27T15:42:20.1523477Z ----------------------------------------------------------------------
2022-09-27T15:42:20.1545357Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:20.1545613Z 
2022-09-27T15:42:20.1545887Z ----------------------------------------------------------------------
2022-09-27T15:42:20.1546122Z Ran 1 test in 0.002s
2022-09-27T15:42:20.1546245Z 
2022-09-27T15:42:20.1546321Z OK (skipped=1)
2022-09-27T15:42:20.1546431Z 
2022-09-27T15:42:20.1546517Z Generating XML reports...
2022-09-27T15:42:20.1575367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154220.xml
2022-09-27T15:42:21.3391430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:21.3391809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:21.3398858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:21.3399208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:21.6779944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:21.6789268Z 
2022-09-27T15:42:21.6789374Z Running tests...
2022-09-27T15:42:21.6789978Z ----------------------------------------------------------------------
2022-09-27T15:42:21.6811306Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:21.6811681Z 
2022-09-27T15:42:21.6812109Z ----------------------------------------------------------------------
2022-09-27T15:42:21.6812437Z Ran 1 test in 0.002s
2022-09-27T15:42:21.6812556Z 
2022-09-27T15:42:21.6812634Z OK (skipped=1)
2022-09-27T15:42:21.6812746Z 
2022-09-27T15:42:21.6812819Z Generating XML reports...
2022-09-27T15:42:21.6842127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154221.xml
2022-09-27T15:42:22.8719716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:22.8720074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:22.8727723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:22.8728087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:23.2137757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:23.2147281Z 
2022-09-27T15:42:23.2147790Z Running tests...
2022-09-27T15:42:23.2148195Z ----------------------------------------------------------------------
2022-09-27T15:42:23.2165444Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:42:23.2165889Z 
2022-09-27T15:42:23.2166319Z ----------------------------------------------------------------------
2022-09-27T15:42:23.2166752Z Ran 1 test in 0.002s
2022-09-27T15:42:23.2166937Z 
2022-09-27T15:42:23.2167034Z OK (skipped=1)
2022-09-27T15:42:23.2167144Z 
2022-09-27T15:42:23.2167231Z Generating XML reports...
2022-09-27T15:42:23.2197305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154223.xml
2022-09-27T15:42:24.4284709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:24.4285170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:24.4292070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:24.4292435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:24.7660316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:24.7669165Z 
2022-09-27T15:42:24.7669263Z Running tests...
2022-09-27T15:42:24.7670343Z ----------------------------------------------------------------------
2022-09-27T15:42:25.0016245Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12079
2022-09-27T15:42:25.0043409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12080
2022-09-27T15:42:25.0070475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12081
2022-09-27T15:42:26.0361333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:26.0361977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:26.0673929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:26.0674914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:26.0850721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:26.0851392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:26.0885309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:26.0885919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:26.1194530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:26.1195208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:26.1233331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:26.1233997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:26.4474558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:26.4475174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:26.4475535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:26.4476170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:26.4476707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:26.4477251Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:26.4483539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:26.4487797Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5484122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:26.5484572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:26.5488588Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5489828Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5494211Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5494919Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5495404Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5501668Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5508084Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5508628Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5512501Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5513170Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5513791Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5518170Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5519118Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5522391Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5525545Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5528510Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5529317Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5531007Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5533502Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5535880Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5538814Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5539766Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5540592Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5544625Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5547354Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5548099Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5550846Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5551603Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5552389Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5555323Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5557465Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5559434Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5562496Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5563280Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5563750Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5566262Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5569800Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5570587Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5572554Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5573279Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5574001Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5579334Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5579810Z 
2022-09-27T15:42:26.5580270Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5582674Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5583402Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5584305Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5587226Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5590592Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5592398Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5592862Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5597004Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5597770Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5599417Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5601562Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5606154Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5614624Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5615426Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5615969Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5616389Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5616828Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5617386Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5618308Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5619411Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5619704Z 
2022-09-27T15:42:26.5623772Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5626573Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5627368Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5630105Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5630840Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5631588Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5634362Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5637186Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5638175Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5641329Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5642113Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5643018Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5646608Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5648208Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5648911Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5650977Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5651692Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5652138Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5655000Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5657865Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5658328Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5667996Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5669041Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5669451Z 
2022-09-27T15:42:26.5669710Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5670156Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5670616Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5671189Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5672119Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5672415Z 
2022-09-27T15:42:26.5676191Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5678301Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5680316Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5682798Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5683545Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5684350Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5686735Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5689372Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5690689Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:26.5692535Z STAGE:2022-09-27 15:42:26 12080:12080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5694225Z STAGE:2022-09-27 15:42:26 12081:12081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.5694996Z STAGE:2022-09-27 15:42:26 12079:12079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:26.9120898Z ok (2.145s)
2022-09-27T15:42:26.9121328Z 
2022-09-27T15:42:26.9121843Z ----------------------------------------------------------------------
2022-09-27T15:42:26.9122173Z Ran 1 test in 2.145s
2022-09-27T15:42:26.9122292Z 
2022-09-27T15:42:26.9122353Z OK
2022-09-27T15:42:26.9122447Z 
2022-09-27T15:42:26.9122540Z Generating XML reports...
2022-09-27T15:42:26.9168848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154224.xml
2022-09-27T15:42:28.2190864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:28.2191243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:28.2199008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:28.2199373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:28.5601075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:28.5610424Z 
2022-09-27T15:42:28.5610899Z Running tests...
2022-09-27T15:42:28.5611322Z ----------------------------------------------------------------------
2022-09-27T15:42:28.7884113Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.227s)
2022-09-27T15:42:28.7884985Z 
2022-09-27T15:42:28.7885335Z ----------------------------------------------------------------------
2022-09-27T15:42:28.7885789Z Ran 1 test in 0.227s
2022-09-27T15:42:28.7886004Z 
2022-09-27T15:42:28.7886137Z OK (skipped=1)
2022-09-27T15:42:28.7886322Z 
2022-09-27T15:42:28.7886483Z Generating XML reports...
2022-09-27T15:42:28.7907286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154228.xml
2022-09-27T15:42:30.0301407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:30.0301786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:30.0310107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:30.0310456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:30.3668377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:30.3677432Z 
2022-09-27T15:42:30.3677560Z Running tests...
2022-09-27T15:42:30.3678499Z ----------------------------------------------------------------------
2022-09-27T15:42:30.6008055Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12145
2022-09-27T15:42:30.6034821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12146
2022-09-27T15:42:30.6061426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12147
2022-09-27T15:42:31.6958119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:31.6958534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:31.7275743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:31.7276152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:31.7292113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:31.7292502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:31.7485884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:31.7486526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:31.7630236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:31.7630891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:31.7825956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:31.7826601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:32.1189375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:32.1290525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:32.1291031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:32.1291999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:32.1292701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:32.1293232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:32.1298145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:32.1299147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:32.1303248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:32.1505737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:42:32.1607357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:42:32.1608107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:42:32.1608716Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:32.1609248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:32.1609771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:32.1612276Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1612650Z 
2022-09-27T15:42:32.1617296Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1628884Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1711827Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1716024Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1716490Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1722778Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1726932Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1727474Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1731922Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1732738Z 
2022-09-27T15:42:32.1735910Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1738011Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1738877Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1742468Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1743317Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1744280Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1746869Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1749478Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1750078Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1754857Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1755643Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1756216Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1757270Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1761270Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1762227Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1764351Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1765106Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1765564Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1769745Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1773316Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1774124Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1776622Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1777398Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1778175Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1782244Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1782829Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1784423Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1788517Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1789460Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1790280Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1791259Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1794942Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1795437Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1799070Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1799920Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1800366Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1803771Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1804584Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1812919Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1813389Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1813986Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1814589Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1815178Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1815652Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1816075Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1818626Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1819781Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1820282Z 
2022-09-27T15:42:32.1823149Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1825263Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1826238Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1828658Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1829140Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1829583Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1834758Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1835704Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1836608Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1843915Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1844366Z 
2022-09-27T15:42:32.1844687Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1845605Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1846202Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1846688Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1858182Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1858853Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1859145Z 
2022-09-27T15:42:32.1859384Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1860731Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1861013Z 
2022-09-27T15:42:32.1861453Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1861900Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1862331Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1865220Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1868200Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1868667Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1878213Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1879182Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1879992Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1880595Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1881176Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1881921Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1882364Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1882800Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1885992Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1886975Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1887862Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1893962Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1896772Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1897227Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1897754Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1902523Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1903216Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1906910Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:32.1907467Z STAGE:2022-09-27 15:42:32 12146:12146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1909757Z STAGE:2022-09-27 15:42:32 12147:12147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.1910206Z STAGE:2022-09-27 15:42:32 12145:12145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:32.6110293Z ok (2.243s)
2022-09-27T15:42:32.6110458Z 
2022-09-27T15:42:32.6111284Z ----------------------------------------------------------------------
2022-09-27T15:42:32.6111568Z Ran 1 test in 2.243s
2022-09-27T15:42:32.6111688Z 
2022-09-27T15:42:32.6111752Z OK
2022-09-27T15:42:32.6111853Z 
2022-09-27T15:42:32.6111948Z Generating XML reports...
2022-09-27T15:42:32.6157413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154230.xml
2022-09-27T15:42:33.9132291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:33.9132690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:33.9139872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:33.9140266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:34.2543742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:34.2553044Z 
2022-09-27T15:42:34.2553139Z Running tests...
2022-09-27T15:42:34.2554151Z ----------------------------------------------------------------------
2022-09-27T15:42:34.4906342Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12210
2022-09-27T15:42:34.4932563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12211
2022-09-27T15:42:34.4959275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12212
2022-09-27T15:42:35.6039642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:35.6040254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:35.6205711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:35.6206407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:35.6243058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:35.6243700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:35.6390584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:35.6391119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:35.6562795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:35.6563446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:35.6594160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:35.6594545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:35.9986425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:36.0086299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:36.0087023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:36.0088001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:36.0088541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:36.0089139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:36.0194066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:36.0194511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:36.0399964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:42:36.0400585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:42:36.1098091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:36.1099907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:42:36.1100694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:36.1107727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:36.1108337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:42:36.1111389Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1111852Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1116789Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1117267Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1122986Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1128801Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1132709Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1133449Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1138735Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1141764Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1145373Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1146022Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1149899Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1153322Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1156330Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1156796Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1160480Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1163067Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1165920Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1166489Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1170016Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1172037Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1174737Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1175187Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1178498Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1182379Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1189576Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1190571Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1191052Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1191795Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1194670Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1195437Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1199582Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1200843Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1203527Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1204009Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1207778Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1210588Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1213809Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1214108Z 
2022-09-27T15:42:36.1219401Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1222460Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1225751Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1226430Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1231542Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1234608Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:42:36.1237432Z STAGE:2022-09-27 15:42:36 12212:12212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.1237877Z STAGE:2022-09-27 15:42:36 12211:12211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:42:36.5007854Z ok (2.245s)
2022-09-27T15:42:36.5008063Z 
2022-09-27T15:42:36.5008569Z ----------------------------------------------------------------------
2022-09-27T15:42:36.5008951Z Ran 1 test in 2.245s
2022-09-27T15:42:36.5009055Z 
2022-09-27T15:42:36.5009117Z OK
2022-09-27T15:42:36.5009211Z 
2022-09-27T15:42:36.5009487Z Generating XML reports...
2022-09-27T15:42:36.5053900Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154234.xml
2022-09-27T15:42:37.8348937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:37.8349313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:37.8355871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:37.8356379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:38.1734760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:38.1744270Z 
2022-09-27T15:42:38.1744462Z Running tests...
2022-09-27T15:42:38.1745069Z ----------------------------------------------------------------------
2022-09-27T15:42:38.4105812Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12271
2022-09-27T15:42:38.4132800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12272
2022-09-27T15:42:38.4159878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12273
2022-09-27T15:42:39.4949412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:39.4950015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:39.5105896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:39.5106284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:39.5282133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:39.5282517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:39.5399236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:39.5399779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:39.5443148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:39.5443838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:39.5717207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:39.5717869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:39.8925026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:39.8925472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:39.8925874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:39.8926878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:39.8927804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:39.8928640Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:39.8932504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:39.8933333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:39.8933909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:40.2204298Z skip: CUDA is not available. (2.046s)
2022-09-27T15:42:40.2204663Z 
2022-09-27T15:42:40.2205070Z ----------------------------------------------------------------------
2022-09-27T15:42:40.2205404Z Ran 1 test in 2.046s
2022-09-27T15:42:40.2205518Z 
2022-09-27T15:42:40.2205592Z OK (skipped=1)
2022-09-27T15:42:40.2205700Z 
2022-09-27T15:42:40.2205789Z Generating XML reports...
2022-09-27T15:42:40.2250465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154238.xml
2022-09-27T15:42:41.5311130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:41.5311506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:41.5319457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:41.5319875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:41.8689546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:41.8698683Z 
2022-09-27T15:42:41.8698812Z Running tests...
2022-09-27T15:42:41.8699406Z ----------------------------------------------------------------------
2022-09-27T15:42:42.0957160Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.225s)
2022-09-27T15:42:42.0957654Z 
2022-09-27T15:42:42.0957845Z ----------------------------------------------------------------------
2022-09-27T15:42:42.0958101Z Ran 1 test in 0.226s
2022-09-27T15:42:42.0958218Z 
2022-09-27T15:42:42.0958291Z OK (skipped=1)
2022-09-27T15:42:42.0958400Z 
2022-09-27T15:42:42.0958486Z Generating XML reports...
2022-09-27T15:42:42.0981805Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154241.xml
2022-09-27T15:42:43.3364845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:43.3365220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:43.3372882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:43.3373225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:43.6749376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:43.6759051Z 
2022-09-27T15:42:43.6759161Z Running tests...
2022-09-27T15:42:43.6759646Z ----------------------------------------------------------------------
2022-09-27T15:42:43.6775556Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:42:43.6776086Z 
2022-09-27T15:42:43.6776403Z ----------------------------------------------------------------------
2022-09-27T15:42:43.6776645Z Ran 1 test in 0.002s
2022-09-27T15:42:43.6776759Z 
2022-09-27T15:42:43.6776833Z OK (skipped=1)
2022-09-27T15:42:43.6776942Z 
2022-09-27T15:42:43.6777028Z Generating XML reports...
2022-09-27T15:42:43.6806027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154243.xml
2022-09-27T15:42:44.8853893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:44.8854267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:44.8861554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:44.8862162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:45.2240572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:45.2249630Z 
2022-09-27T15:42:45.2249772Z Running tests...
2022-09-27T15:42:45.2250179Z ----------------------------------------------------------------------
2022-09-27T15:42:45.2266195Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:42:45.2266535Z 
2022-09-27T15:42:45.2266747Z ----------------------------------------------------------------------
2022-09-27T15:42:45.2267059Z Ran 1 test in 0.002s
2022-09-27T15:42:45.2267160Z 
2022-09-27T15:42:45.2267234Z OK (skipped=1)
2022-09-27T15:42:45.2267344Z 
2022-09-27T15:42:45.2267427Z Generating XML reports...
2022-09-27T15:42:45.2296213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154245.xml
2022-09-27T15:42:46.4213371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:46.4214018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:46.4222328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:46.4222964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:46.7621426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:46.7631554Z 
2022-09-27T15:42:46.7631950Z Running tests...
2022-09-27T15:42:46.7632377Z ----------------------------------------------------------------------
2022-09-27T15:42:47.0003223Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12354
2022-09-27T15:42:47.0029748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12355
2022-09-27T15:42:47.0056942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12356
2022-09-27T15:42:48.0569178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:48.0569564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:48.0757903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:48.0758432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:48.0904825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:48.0905494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:48.0971441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:48.0972053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:48.1105970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:48.1106628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:48.1297440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:48.1298075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:48.4483312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:48.4585290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:48.4592848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:48.4593538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:48.4685615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:48.4687719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:48.4699049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:48.5699370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:48.5699985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:48.9110545Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:42:48.9110844Z 
2022-09-27T15:42:48.9111373Z ----------------------------------------------------------------------
2022-09-27T15:42:48.9111649Z Ran 1 test in 2.148s
2022-09-27T15:42:48.9111808Z 
2022-09-27T15:42:48.9111882Z OK (skipped=1)
2022-09-27T15:42:48.9112198Z 
2022-09-27T15:42:48.9112398Z Generating XML reports...
2022-09-27T15:42:48.9157960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154246.xml
2022-09-27T15:42:50.2220364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:50.2221005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:50.2229278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:50.2229939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:50.5620514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:50.5630889Z 
2022-09-27T15:42:50.5631187Z Running tests...
2022-09-27T15:42:50.5631818Z ----------------------------------------------------------------------
2022-09-27T15:42:50.7992583Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12407
2022-09-27T15:42:50.8018866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12408
2022-09-27T15:42:50.8045773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12409
2022-09-27T15:42:51.8741710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:51.8742316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:51.8974614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:51.8974988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:51.9054726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:51.9055076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:51.9076170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:51.9076546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:51.9326137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:51.9326505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:51.9394973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:51.9395456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:52.2658718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:52.2659264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:52.2659693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:52.2660320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:52.2660851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:52.2661376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:52.2765218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:52.2765874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:52.3669929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:52.6095114Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:42:52.6095451Z 
2022-09-27T15:42:52.6095956Z ----------------------------------------------------------------------
2022-09-27T15:42:52.6096216Z Ran 1 test in 2.046s
2022-09-27T15:42:52.6096332Z 
2022-09-27T15:42:52.6096407Z OK (skipped=1)
2022-09-27T15:42:52.6096517Z 
2022-09-27T15:42:52.6096604Z Generating XML reports...
2022-09-27T15:42:52.6141613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154250.xml
2022-09-27T15:42:53.9760042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:53.9760423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:53.9767899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:53.9768273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:54.3318457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:54.3328004Z 
2022-09-27T15:42:54.3328433Z Running tests...
2022-09-27T15:42:54.3328811Z ----------------------------------------------------------------------
2022-09-27T15:42:54.5718131Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12460
2022-09-27T15:42:54.5746186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12461
2022-09-27T15:42:54.5773595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12462
2022-09-27T15:42:55.6066659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:55.6067365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:55.6236265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:55.6236790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:55.6398792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:55.6399402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:55.6577191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:55.6577847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:55.6654054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:55.6655014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:55.7005507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:55.7006436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:56.0213156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:56.0314611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:56.0315357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:56.0315987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:56.0316510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:56.0317051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:56.0322661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:56.0323695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:56.0324201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:42:56.3818994Z skip: Need at least 2 CUDA devices (2.049s)
2022-09-27T15:42:56.3819252Z 
2022-09-27T15:42:56.3819767Z ----------------------------------------------------------------------
2022-09-27T15:42:56.3820200Z Ran 1 test in 2.049s
2022-09-27T15:42:56.3820404Z 
2022-09-27T15:42:56.3820533Z OK (skipped=1)
2022-09-27T15:42:56.3820727Z 
2022-09-27T15:42:56.3820863Z Generating XML reports...
2022-09-27T15:42:56.3866211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154254.xml
2022-09-27T15:42:57.7125254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:57.7125913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:57.7134038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:57.7134680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:58.0525708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:42:58.0535891Z 
2022-09-27T15:42:58.0536350Z Running tests...
2022-09-27T15:42:58.0536742Z ----------------------------------------------------------------------
2022-09-27T15:42:58.2888820Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12513
2022-09-27T15:42:58.2915747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12514
2022-09-27T15:42:58.2942655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12515
2022-09-27T15:42:59.3730576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:59.3731241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:59.3865340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:59.3865822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:59.3898488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:42:59.3898878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:42:59.4075943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:59.4076341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:59.4221312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:59.4221908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:59.4251578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:42:59.4251945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:42:59.7617642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:42:59.7718301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:42:59.7719046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:42:59.7719828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:59.7720361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:59.7720898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:42:59.7824657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:42:59.8731587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:42:59.8732237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:00.1992741Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:43:00.1993028Z 
2022-09-27T15:43:00.1993470Z ----------------------------------------------------------------------
2022-09-27T15:43:00.1993877Z Ran 1 test in 2.146s
2022-09-27T15:43:00.1994063Z 
2022-09-27T15:43:00.1994154Z OK (skipped=1)
2022-09-27T15:43:00.1994323Z 
2022-09-27T15:43:00.1994455Z Generating XML reports...
2022-09-27T15:43:00.2039923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154258.xml
2022-09-27T15:43:01.5354989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:01.5355366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:01.5363463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:01.5363835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:01.8795481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:01.8805130Z 
2022-09-27T15:43:01.8805252Z Running tests...
2022-09-27T15:43:01.8805773Z ----------------------------------------------------------------------
2022-09-27T15:43:01.8835698Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-09-27T15:43:01.8836173Z 
2022-09-27T15:43:01.8836458Z ----------------------------------------------------------------------
2022-09-27T15:43:01.8836738Z Ran 1 test in 0.003s
2022-09-27T15:43:01.8836854Z 
2022-09-27T15:43:01.8836940Z OK (skipped=1)
2022-09-27T15:43:01.8837096Z 
2022-09-27T15:43:01.8837182Z Generating XML reports...
2022-09-27T15:43:01.8868125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154301.xml
2022-09-27T15:43:03.0745393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:03.0745817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:03.0754133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:03.0754503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:03.4142922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:03.4152816Z 
2022-09-27T15:43:03.4152973Z Running tests...
2022-09-27T15:43:03.4153494Z ----------------------------------------------------------------------
2022-09-27T15:43:03.6523098Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12576
2022-09-27T15:43:03.6549992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12577
2022-09-27T15:43:03.6576941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12578
2022-09-27T15:43:04.7439949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:04.7440330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:04.7756372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:04.7757018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:04.7934013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:04.7934376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:04.8025557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:04.8025929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:04.8287666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:04.8288180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:04.8374104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:04.8374757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:05.1548971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:05.1648216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:05.1648951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:05.1649899Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:05.1650743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:05.1651282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:05.1756234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:05.1756786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:05.2661509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:05.5626954Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:43:05.5627694Z 
2022-09-27T15:43:05.5628156Z ----------------------------------------------------------------------
2022-09-27T15:43:05.5628461Z Ran 1 test in 2.147s
2022-09-27T15:43:05.5628574Z 
2022-09-27T15:43:05.5628648Z OK (skipped=1)
2022-09-27T15:43:05.5628757Z 
2022-09-27T15:43:05.5628842Z Generating XML reports...
2022-09-27T15:43:05.5674623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154303.xml
2022-09-27T15:43:06.8904649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:06.8905721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:06.8913949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:06.8914591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:07.2299940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:07.2310273Z 
2022-09-27T15:43:07.2310919Z Running tests...
2022-09-27T15:43:07.2311366Z ----------------------------------------------------------------------
2022-09-27T15:43:07.4683821Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12629
2022-09-27T15:43:07.4709988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12630
2022-09-27T15:43:07.4736655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12631
2022-09-27T15:43:08.5499290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:08.5499682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:08.5603361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:08.5603936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:08.5835600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:08.5836048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:08.5947472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:08.5948116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:08.6083198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:08.6083825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:08.6399193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:08.6399841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:08.9733560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:08.9831696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:08.9832167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:08.9832785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:08.9833326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:08.9834280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:08.9938824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:08.9939741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:09.0847484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:09.3783778Z skip: Need at least 3 CUDA devices (2.147s)
2022-09-27T15:43:09.3784138Z 
2022-09-27T15:43:09.3784616Z ----------------------------------------------------------------------
2022-09-27T15:43:09.3784871Z Ran 1 test in 2.147s
2022-09-27T15:43:09.3784974Z 
2022-09-27T15:43:09.3785048Z OK (skipped=1)
2022-09-27T15:43:09.3785287Z 
2022-09-27T15:43:09.3785373Z Generating XML reports...
2022-09-27T15:43:09.3831498Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154307.xml
2022-09-27T15:43:10.6898946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:10.6899326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:10.6907100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:10.6907655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:11.0333758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:11.0343165Z 
2022-09-27T15:43:11.0343325Z Running tests...
2022-09-27T15:43:11.0344044Z ----------------------------------------------------------------------
2022-09-27T15:43:11.0384159Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s)
2022-09-27T15:43:11.0384681Z 
2022-09-27T15:43:11.0384960Z ----------------------------------------------------------------------
2022-09-27T15:43:11.0385285Z Ran 1 test in 0.004s
2022-09-27T15:43:11.0385400Z 
2022-09-27T15:43:11.0385460Z OK (skipped=1)
2022-09-27T15:43:11.0385569Z 
2022-09-27T15:43:11.0385656Z Generating XML reports...
2022-09-27T15:43:11.0415171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154311.xml
2022-09-27T15:43:12.2643605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:12.2643990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:12.2651751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:12.2652170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:12.6038980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:12.6048536Z 
2022-09-27T15:43:12.6048649Z Running tests...
2022-09-27T15:43:12.6049088Z ----------------------------------------------------------------------
2022-09-27T15:43:12.6083211Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-09-27T15:43:12.6083552Z 
2022-09-27T15:43:12.6083933Z ----------------------------------------------------------------------
2022-09-27T15:43:12.6084347Z Ran 1 test in 0.003s
2022-09-27T15:43:12.6084474Z 
2022-09-27T15:43:12.6084555Z OK (skipped=1)
2022-09-27T15:43:12.6084664Z 
2022-09-27T15:43:12.6084747Z Generating XML reports...
2022-09-27T15:43:12.6114577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154312.xml
2022-09-27T15:43:13.8273041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:13.8273418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:13.8280110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:13.8280469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:14.1656729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:14.1666448Z 
2022-09-27T15:43:14.1666729Z Running tests...
2022-09-27T15:43:14.1667342Z ----------------------------------------------------------------------
2022-09-27T15:43:14.4021145Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12702
2022-09-27T15:43:14.4047461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12703
2022-09-27T15:43:14.4075102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12704
2022-09-27T15:43:15.4312026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:15.4312752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:15.4620471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:15.4621083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:15.4822869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:15.4823601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:15.4971992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:15.4972402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:15.5158935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:15.5159471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:15.5306275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:15.5306929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:15.8540665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:15.8640689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:15.8641212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:15.8642075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:15.8642663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:15.8643193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:15.8747697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:15.8807396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99196eh2
2022-09-27T15:43:15.8808566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99196eh2/_remote_module_non_scriptable.py
2022-09-27T15:43:15.9650350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:15.9652652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:15.9713109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmxvpwow
2022-09-27T15:43:15.9715161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmxvpwow/_remote_module_non_scriptable.py
2022-09-27T15:43:15.9717152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_01k8bxr
2022-09-27T15:43:15.9719225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_01k8bxr/_remote_module_non_scriptable.py
2022-09-27T15:43:15.9831031Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9832301Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9833824Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9835684Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:43:15.9836905Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:43:15.9838182Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:43:15.9839171Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:43:15.9840900Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:43:15.9841974Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:43:15.9842603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:43:15.9843128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:43:15.9844175Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9845379Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9846099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:43:15.9848303Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9853097Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9854808Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9855967Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9860049Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9861625Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9862768Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9865952Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9867391Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9868541Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9871976Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9873393Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:15.9874643Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:43:16.3123007Z ok (2.145s)
2022-09-27T15:43:16.3123263Z 
2022-09-27T15:43:16.3123747Z ----------------------------------------------------------------------
2022-09-27T15:43:16.3124184Z Ran 1 test in 2.146s
2022-09-27T15:43:16.3124379Z 
2022-09-27T15:43:16.3124481Z OK
2022-09-27T15:43:16.3124630Z 
2022-09-27T15:43:16.3124778Z Generating XML reports...
2022-09-27T15:43:16.3170193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154314.xml
2022-09-27T15:43:17.6129879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:17.6130306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:17.6138140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:17.6138524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:17.9509373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:17.9519142Z 
2022-09-27T15:43:17.9519276Z Running tests...
2022-09-27T15:43:17.9519868Z ----------------------------------------------------------------------
2022-09-27T15:43:17.9562773Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s)
2022-09-27T15:43:17.9563100Z 
2022-09-27T15:43:17.9563512Z ----------------------------------------------------------------------
2022-09-27T15:43:17.9563882Z Ran 1 test in 0.004s
2022-09-27T15:43:17.9563995Z 
2022-09-27T15:43:17.9564055Z OK (skipped=1)
2022-09-27T15:43:17.9564164Z 
2022-09-27T15:43:17.9564248Z Generating XML reports...
2022-09-27T15:43:17.9593663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154317.xml
2022-09-27T15:43:19.1670071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:19.1670491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:19.1678413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:19.1678777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:19.5050540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:19.5060425Z 
2022-09-27T15:43:19.5060520Z Running tests...
2022-09-27T15:43:19.5061580Z ----------------------------------------------------------------------
2022-09-27T15:43:19.5089256Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-09-27T15:43:19.5089662Z 
2022-09-27T15:43:19.5090262Z ----------------------------------------------------------------------
2022-09-27T15:43:19.5090753Z Ran 1 test in 0.003s
2022-09-27T15:43:19.5090913Z 
2022-09-27T15:43:19.5090998Z OK (skipped=1)
2022-09-27T15:43:19.5091174Z 
2022-09-27T15:43:19.5091261Z Generating XML reports...
2022-09-27T15:43:19.5121342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154319.xml
2022-09-27T15:43:20.7278753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:20.7279426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:20.7287978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:20.7288624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:21.0662022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:21.0671729Z 
2022-09-27T15:43:21.0672140Z Running tests...
2022-09-27T15:43:21.0672543Z ----------------------------------------------------------------------
2022-09-27T15:43:21.3064112Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12775
2022-09-27T15:43:21.3090097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12776
2022-09-27T15:43:21.3117288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12777
2022-09-27T15:43:22.3862350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:22.3862994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:22.4062842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:22.4063244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:22.4195943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:22.4196592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:22.4258718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:22.4259331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:22.4411954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:22.4412342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:22.4598961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:22.4599380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:22.7863114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:22.7963588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:22.7964107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:22.7964754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:22.7965290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:22.7965823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:22.7972010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:22.7973467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:22.7973886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:23.1165071Z skip: Need at least 2 CUDA devices (2.049s)
2022-09-27T15:43:23.1165257Z 
2022-09-27T15:43:23.1165572Z ----------------------------------------------------------------------
2022-09-27T15:43:23.1165826Z Ran 1 test in 2.049s
2022-09-27T15:43:23.1165927Z 
2022-09-27T15:43:23.1166001Z OK (skipped=1)
2022-09-27T15:43:23.1166110Z 
2022-09-27T15:43:23.1166197Z Generating XML reports...
2022-09-27T15:43:23.1210843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154321.xml
2022-09-27T15:43:24.4003576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:24.4004232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:24.4012725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:24.4013355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:24.7379917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:24.7390396Z 
2022-09-27T15:43:24.7390832Z Running tests...
2022-09-27T15:43:24.7391263Z ----------------------------------------------------------------------
2022-09-27T15:43:24.9729566Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12828
2022-09-27T15:43:24.9756589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12829
2022-09-27T15:43:24.9783685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12830
2022-09-27T15:43:26.0487798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:26.0488173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:26.0680563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:26.0680984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:26.0684993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:26.0685499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:26.0823548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:26.0824049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:26.1045018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:26.1045589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:26.1046836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:26.1047487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:26.4483521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:26.4585058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:26.4585563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:26.4586246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:26.4587022Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:26.4588090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:26.4592753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:26.4593588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:26.4595717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:26.7835327Z skip: Need at least 3 CUDA devices (2.044s)
2022-09-27T15:43:26.7835639Z 
2022-09-27T15:43:26.7836119Z ----------------------------------------------------------------------
2022-09-27T15:43:26.7836373Z Ran 1 test in 2.044s
2022-09-27T15:43:26.7836491Z 
2022-09-27T15:43:26.7836565Z OK (skipped=1)
2022-09-27T15:43:26.7836677Z 
2022-09-27T15:43:26.7836765Z Generating XML reports...
2022-09-27T15:43:26.7882221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154324.xml
2022-09-27T15:43:28.0725208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:28.0725599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:28.0733534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:28.0733955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:28.4142900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:28.4153537Z 
2022-09-27T15:43:28.4153768Z Running tests...
2022-09-27T15:43:28.6507190Z ----------------------------------------------------------------------
2022-09-27T15:43:28.6508006Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12881
2022-09-27T15:43:28.6534604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12882
2022-09-27T15:43:28.6561490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12883
2022-09-27T15:43:29.7381185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:29.7381856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:29.7505699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:29.7506183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:29.7662337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:29.7731232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:29.7732094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:29.7732687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:29.7860412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:29.7861106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:29.8002766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:29.8003406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:30.1266080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:30.1366325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:30.1366808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:30.1367640Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:30.1368188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:30.1368774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:30.1472463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:30.2378531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:30.2379434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:30.5610854Z skip: Need at least 3 CUDA devices (2.145s)
2022-09-27T15:43:30.5611124Z 
2022-09-27T15:43:30.5611475Z ----------------------------------------------------------------------
2022-09-27T15:43:30.5611774Z Ran 1 test in 2.146s
2022-09-27T15:43:30.5611940Z 
2022-09-27T15:43:30.5612024Z OK (skipped=1)
2022-09-27T15:43:30.5612121Z 
2022-09-27T15:43:30.5612210Z Generating XML reports...
2022-09-27T15:43:30.5657235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154328.xml
2022-09-27T15:43:31.8761523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:31.8761883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:31.8769269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:31.8769644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:32.2144328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:32.2154028Z 
2022-09-27T15:43:32.2154664Z Running tests...
2022-09-27T15:43:32.2155108Z ----------------------------------------------------------------------
2022-09-27T15:43:32.4501542Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12934
2022-09-27T15:43:32.4527345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12935
2022-09-27T15:43:32.4554610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12936
2022-09-27T15:43:33.5365217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:33.5365843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:33.5404539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:33.5405212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:33.5422645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:33.5423286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:33.5725174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:33.5725845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:33.5767112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:33.5767842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:33.5788879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:33.5789466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:33.9085056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:33.9085686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:33.9086201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:33.9087216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:33.9087786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:33.9088298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:33.9190621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:34.0097781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:34.0098258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:34.2601408Z skip: Need at least 3 CUDA devices (2.044s)
2022-09-27T15:43:34.2601643Z 
2022-09-27T15:43:34.2601951Z ----------------------------------------------------------------------
2022-09-27T15:43:34.2602201Z Ran 1 test in 2.045s
2022-09-27T15:43:34.2602303Z 
2022-09-27T15:43:34.2602376Z OK (skipped=1)
2022-09-27T15:43:34.2602484Z 
2022-09-27T15:43:34.2610210Z Generating XML reports...
2022-09-27T15:43:34.2648773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154332.xml
2022-09-27T15:43:35.5548849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:35.5549512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:35.5557814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:35.5558466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:35.8928101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:35.8938082Z 
2022-09-27T15:43:35.8938390Z Running tests...
2022-09-27T15:43:35.8939012Z ----------------------------------------------------------------------
2022-09-27T15:43:36.1302169Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12987
2022-09-27T15:43:36.1327948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12988
2022-09-27T15:43:36.1355220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12989
2022-09-27T15:43:37.1949814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:37.1950214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:37.2173315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:37.2173786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:37.2272253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:37.2272691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:37.2510638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:37.2511283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:37.2614659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:37.2615555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:37.2925507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:37.2926398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:37.6089780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:37.6191902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:37.6192316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:37.6192937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:37.6193477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:37.6194022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:37.6297138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:37.7205494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:37.7206096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:38.0403937Z skip: Need at least 3 CUDA devices (2.146s)
2022-09-27T15:43:38.0404147Z 
2022-09-27T15:43:38.0404563Z ----------------------------------------------------------------------
2022-09-27T15:43:38.0404818Z Ran 1 test in 2.147s
2022-09-27T15:43:38.0404934Z 
2022-09-27T15:43:38.0404995Z OK (skipped=1)
2022-09-27T15:43:38.0405106Z 
2022-09-27T15:43:38.0405193Z Generating XML reports...
2022-09-27T15:43:38.0450756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154335.xml
2022-09-27T15:43:39.3668992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:39.3669366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:39.3676620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:39.3677002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:39.7042439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:39.7051627Z 
2022-09-27T15:43:39.7051725Z Running tests...
2022-09-27T15:43:39.7052191Z ----------------------------------------------------------------------
2022-09-27T15:43:39.9403814Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13040
2022-09-27T15:43:39.9430115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13041
2022-09-27T15:43:39.9456891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13042
2022-09-27T15:43:40.9915110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:40.9915811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:40.9955559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:40.9955922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:41.0148382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:41.0148746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:41.0269367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:41.0270181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:41.0318317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:41.0319094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:41.0544175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:41.0544841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:41.3823025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:41.3823602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:41.3824157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:41.3824844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:41.3825455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:41.3825985Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:41.3928764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:41.4834658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:41.4835230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:41.7505818Z skip: Need at least 3 CUDA devices (2.045s)
2022-09-27T15:43:41.7506086Z 
2022-09-27T15:43:41.7506598Z ----------------------------------------------------------------------
2022-09-27T15:43:41.7507058Z Ran 1 test in 2.045s
2022-09-27T15:43:41.7507288Z 
2022-09-27T15:43:41.7507421Z OK (skipped=1)
2022-09-27T15:43:41.7507608Z 
2022-09-27T15:43:41.7507752Z Generating XML reports...
2022-09-27T15:43:41.7553908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154339.xml
2022-09-27T15:43:43.0399384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:43.0399765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:43.0407753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:43.0408221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:43.3806370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:43.3816540Z 
2022-09-27T15:43:43.3817060Z Running tests...
2022-09-27T15:43:43.3817476Z ----------------------------------------------------------------------
2022-09-27T15:43:43.6157908Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13093
2022-09-27T15:43:43.6183794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13094
2022-09-27T15:43:43.6210872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13095
2022-09-27T15:43:44.7259973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:44.7578509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:44.7579474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:44.7580058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:44.7592975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:44.7612672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:44.7613772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:44.7614379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:44.7947886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:44.7948619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:44.7965321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:44.7965711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:45.1433803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:45.1434517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:45.1434898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:45.1435528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:45.1436051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:45.1436582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:45.1441226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:45.1442936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:45.1443621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:45.4260039Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:43:45.4260336Z 
2022-09-27T15:43:45.4260723Z ----------------------------------------------------------------------
2022-09-27T15:43:45.4260996Z Ran 1 test in 2.044s
2022-09-27T15:43:45.4261112Z 
2022-09-27T15:43:45.4261189Z OK (skipped=1)
2022-09-27T15:43:45.4261300Z 
2022-09-27T15:43:45.4261375Z Generating XML reports...
2022-09-27T15:43:45.4305400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154343.xml
2022-09-27T15:43:46.7087130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:46.7087506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:46.7095590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:46.7096100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:47.0462916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:47.0472691Z 
2022-09-27T15:43:47.0472818Z Running tests...
2022-09-27T15:43:47.0473427Z ----------------------------------------------------------------------
2022-09-27T15:43:47.2845272Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13146
2022-09-27T15:43:47.2871902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13147
2022-09-27T15:43:47.2899328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13148
2022-09-27T15:43:48.3479656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:48.3480589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:48.3677992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:48.3678944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:48.3810929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:48.3811328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:48.3816909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:48.3817417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:48.4035516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:48.4035951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:48.4158481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:48.7589726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:48.7590152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:48.7690757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:48.7691479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:48.7692124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:48.7692662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:48.7693203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:48.7699406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:48.7700036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:48.7700403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:49.0947095Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T15:43:49.0947385Z 
2022-09-27T15:43:49.0947887Z ----------------------------------------------------------------------
2022-09-27T15:43:49.0948290Z Ran 1 test in 2.047s
2022-09-27T15:43:49.0948407Z 
2022-09-27T15:43:49.0948484Z OK (skipped=1)
2022-09-27T15:43:49.0948593Z 
2022-09-27T15:43:49.0948666Z Generating XML reports...
2022-09-27T15:43:49.0995079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154347.xml
2022-09-27T15:43:50.3818180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:50.3818558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:50.3826579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:50.3827119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:50.7213432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:50.7223614Z 
2022-09-27T15:43:50.7223737Z Running tests...
2022-09-27T15:43:50.7224474Z ----------------------------------------------------------------------
2022-09-27T15:43:50.9587047Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13199
2022-09-27T15:43:50.9613410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13200
2022-09-27T15:43:50.9640034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13201
2022-09-27T15:43:52.0766262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:52.0854983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:52.0855943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:52.0856574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:52.1109618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:52.1110259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:52.1196708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:52.1197341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:52.1527243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:52.1527618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:52.1842795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:52.1843185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:52.5204140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:52.5304730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:52.5305294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:52.5305935Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:52.5306493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:52.5307006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:52.5412251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:52.5414016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:52.6318110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:52.8687129Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T15:43:52.8687379Z 
2022-09-27T15:43:52.8687992Z ----------------------------------------------------------------------
2022-09-27T15:43:52.8688430Z Ran 1 test in 2.146s
2022-09-27T15:43:52.8688545Z 
2022-09-27T15:43:52.8688619Z OK (skipped=1)
2022-09-27T15:43:52.8688719Z 
2022-09-27T15:43:52.8688812Z Generating XML reports...
2022-09-27T15:43:52.8734496Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154350.xml
2022-09-27T15:43:54.1845238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:54.1845609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:54.1852972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:54.1853465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:54.5270369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:54.5279787Z 
2022-09-27T15:43:54.5280393Z Running tests...
2022-09-27T15:43:54.5280845Z ----------------------------------------------------------------------
2022-09-27T15:43:54.7625673Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13252
2022-09-27T15:43:54.7650369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13253
2022-09-27T15:43:54.7677780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13254
2022-09-27T15:43:55.8124967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:55.8125703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:55.8252862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:55.8253415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:55.8300623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:55.8301166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:55.8476608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:55.8477236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:55.8607553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:55.8608196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:55.8653226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:55.8653902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:56.1935085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:56.2035851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:56.2036568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:56.2037447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:56.2037988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:56.2038510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:56.2043049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:43:56.2046338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:56.2046942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:56.4725944Z skip: Need at least 2 CUDA devices (1.944s)
2022-09-27T15:43:56.4726254Z 
2022-09-27T15:43:56.4726759Z ----------------------------------------------------------------------
2022-09-27T15:43:56.4727006Z Ran 1 test in 1.945s
2022-09-27T15:43:56.4727121Z 
2022-09-27T15:43:56.4727193Z OK (skipped=1)
2022-09-27T15:43:56.4727299Z 
2022-09-27T15:43:56.4727386Z Generating XML reports...
2022-09-27T15:43:56.4774265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154354.xml
2022-09-27T15:43:57.7687841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:57.7688228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:57.7695989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:57.7696417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:58.1082172Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:43:58.1092359Z 
2022-09-27T15:43:58.1092573Z Running tests...
2022-09-27T15:43:58.1093022Z ----------------------------------------------------------------------
2022-09-27T15:43:58.3476668Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13305
2022-09-27T15:43:58.3502840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13306
2022-09-27T15:43:58.3531533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13307
2022-09-27T15:43:59.4407627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:59.4408256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:59.4429806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:59.4430455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:59.4698060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:43:59.4698709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:43:59.4753578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:59.4754324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:59.4782743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:59.4783321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:59.5029198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:43:59.5029857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:43:59.8241183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:43:59.8291232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:43:59.8291958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:43:59.8292592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:59.8293121Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:59.8341953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:43:59.8397790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:43:59.8398274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:43:59.9355172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:00.2580968Z skip: Need at least 2 CUDA devices (2.148s)
2022-09-27T15:44:00.2581214Z 
2022-09-27T15:44:00.2581519Z ----------------------------------------------------------------------
2022-09-27T15:44:00.2581774Z Ran 1 test in 2.149s
2022-09-27T15:44:00.2581891Z 
2022-09-27T15:44:00.2581965Z OK (skipped=1)
2022-09-27T15:44:00.2582078Z 
2022-09-27T15:44:00.2582368Z Generating XML reports...
2022-09-27T15:44:00.2628281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154358.xml
2022-09-27T15:44:01.5710214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:01.5710841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:01.5718869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:01.5719518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:01.9122120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:01.9132072Z 
2022-09-27T15:44:01.9132404Z Running tests...
2022-09-27T15:44:01.9133009Z ----------------------------------------------------------------------
2022-09-27T15:44:02.1502090Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13358
2022-09-27T15:44:02.1528038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13359
2022-09-27T15:44:02.1554915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13360
2022-09-27T15:44:03.2522363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:03.2523023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:03.2541719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:03.2542108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:03.2547463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:03.2547820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:03.2880535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:03.2880940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:03.2899641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:03.2900179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:03.2908598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:03.2909227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:03.6080377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:03.6180655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:03.6181125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:03.6181769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:03.6182305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:03.6182833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:03.6287230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:03.6288382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:03.7194285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:03.9603337Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T15:44:03.9603667Z 
2022-09-27T15:44:03.9604166Z ----------------------------------------------------------------------
2022-09-27T15:44:03.9604711Z Ran 1 test in 2.047s
2022-09-27T15:44:03.9604903Z 
2022-09-27T15:44:03.9605041Z OK (skipped=1)
2022-09-27T15:44:03.9605167Z 
2022-09-27T15:44:03.9605253Z Generating XML reports...
2022-09-27T15:44:03.9650030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154401.xml
2022-09-27T15:44:05.2507046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:05.2507690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:05.2515999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:05.2516641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:05.5907710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:05.5917626Z 
2022-09-27T15:44:05.5917927Z Running tests...
2022-09-27T15:44:05.5918516Z ----------------------------------------------------------------------
2022-09-27T15:44:05.8272141Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13411
2022-09-27T15:44:05.8299296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13412
2022-09-27T15:44:05.8328802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13413
2022-09-27T15:44:06.8617914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:06.8618310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:06.8772753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:06.8773449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:06.8924141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:06.8924673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:06.9026118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:06.9026716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:06.9142257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:06.9142900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:06.9262149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:06.9262786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:07.2626206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:07.2626968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:07.2627518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:07.2628148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:07.2628683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:07.2629384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:07.2730998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:07.3639801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:07.3640370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:07.6376126Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:44:07.6376332Z 
2022-09-27T15:44:07.6376736Z ----------------------------------------------------------------------
2022-09-27T15:44:07.6377191Z Ran 1 test in 2.046s
2022-09-27T15:44:07.6377389Z 
2022-09-27T15:44:07.6377528Z OK (skipped=1)
2022-09-27T15:44:07.6377687Z 
2022-09-27T15:44:07.6377762Z Generating XML reports...
2022-09-27T15:44:07.6422850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154405.xml
2022-09-27T15:44:08.9624085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:08.9624476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:08.9632133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:08.9632483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:09.3005574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:09.3015550Z 
2022-09-27T15:44:09.3015872Z Running tests...
2022-09-27T15:44:09.3016480Z ----------------------------------------------------------------------
2022-09-27T15:44:09.5387421Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13464
2022-09-27T15:44:09.5413536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13465
2022-09-27T15:44:09.5440628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13466
2022-09-27T15:44:10.6014908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:10.6015508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:10.6023588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:10.6024113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:10.6364748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:10.6365136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:10.6371680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:10.6372299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:10.6392036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:10.6392642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:10.6707972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:10.6708642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:11.0174580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:11.0274286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:11.0285233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:11.0286515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:11.0376413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:11.0377152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:11.0393510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:11.1389394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:11.1389868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:11.4491622Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:44:11.4491906Z 
2022-09-27T15:44:11.4492357Z ----------------------------------------------------------------------
2022-09-27T15:44:11.4492758Z Ran 1 test in 2.147s
2022-09-27T15:44:11.4492938Z 
2022-09-27T15:44:11.4493062Z OK (skipped=1)
2022-09-27T15:44:11.4493214Z 
2022-09-27T15:44:11.4493353Z Generating XML reports...
2022-09-27T15:44:11.4539092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154409.xml
2022-09-27T15:44:12.7614209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:12.7614653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:12.7621766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:12.7622193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:13.1026199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:13.1035299Z 
2022-09-27T15:44:13.1035406Z Running tests...
2022-09-27T15:44:13.1036234Z ----------------------------------------------------------------------
2022-09-27T15:44:13.3382363Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13517
2022-09-27T15:44:13.3408605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13518
2022-09-27T15:44:13.3435667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13519
2022-09-27T15:44:14.4561072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:14.4561687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:14.4882575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:14.4883239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:14.4980364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:14.4981067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:14.5077481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:14.5077868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:14.5322801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:14.5323420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:14.5417445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:14.5418065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:14.8700346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:14.8800707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:14.8801540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:14.8802308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:14.8803198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:14.8803928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:14.8907549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:14.9813866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:14.9814469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:15.2486824Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:44:15.2487987Z 
2022-09-27T15:44:15.2488420Z ----------------------------------------------------------------------
2022-09-27T15:44:15.2488863Z Ran 1 test in 2.145s
2022-09-27T15:44:15.2489058Z 
2022-09-27T15:44:15.2489186Z OK (skipped=1)
2022-09-27T15:44:15.2489372Z 
2022-09-27T15:44:15.2489517Z Generating XML reports...
2022-09-27T15:44:15.2537036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154413.xml
2022-09-27T15:44:16.5474987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:16.5475354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:16.5483300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:16.5483668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:16.8855040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:16.8868439Z 
2022-09-27T15:44:16.8868740Z Running tests...
2022-09-27T15:44:16.8869198Z ----------------------------------------------------------------------
2022-09-27T15:44:17.1220750Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13570
2022-09-27T15:44:17.1245617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13571
2022-09-27T15:44:17.1273788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13572
2022-09-27T15:44:18.2197695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:18.2198312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:18.2420115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:18.2420499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:18.2559358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:18.2559737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:18.2567825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:18.2568209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:18.2786636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:18.2787010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:18.2897654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:18.2898175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:18.6172913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:18.6273810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:18.6274384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:18.6275272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:18.6276099Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:18.6276629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:18.6283731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:18.6286035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:18.6286587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:18.9322246Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:44:18.9322544Z 
2022-09-27T15:44:18.9322955Z ----------------------------------------------------------------------
2022-09-27T15:44:18.9323284Z Ran 1 test in 2.045s
2022-09-27T15:44:18.9323404Z 
2022-09-27T15:44:18.9323479Z OK (skipped=1)
2022-09-27T15:44:18.9323576Z 
2022-09-27T15:44:18.9323665Z Generating XML reports...
2022-09-27T15:44:18.9368853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154416.xml
2022-09-27T15:44:20.2358111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:20.2358585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:20.2365821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:20.2366259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:20.5750340Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:20.5759449Z 
2022-09-27T15:44:20.5759729Z Running tests...
2022-09-27T15:44:20.5760463Z ----------------------------------------------------------------------
2022-09-27T15:44:20.8095943Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13623
2022-09-27T15:44:20.8122462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13624
2022-09-27T15:44:20.8149021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13625
2022-09-27T15:44:21.8761037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:21.8761655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:21.8918900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:21.9058163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:21.9059005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:21.9059504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:21.9098181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:21.9098724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:21.9276875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:21.9277502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:21.9396932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:21.9397547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:22.2581930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:22.2675664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:22.2676125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:22.2676744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:22.2677270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:22.2682792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:22.2783128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:22.2784043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:22.3695736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:22.6197301Z skip: Need at least 2 CUDA devices (2.043s)
2022-09-27T15:44:22.6197534Z 
2022-09-27T15:44:22.6197996Z ----------------------------------------------------------------------
2022-09-27T15:44:22.6198433Z Ran 1 test in 2.044s
2022-09-27T15:44:22.6198645Z 
2022-09-27T15:44:22.6198782Z OK (skipped=1)
2022-09-27T15:44:22.6198916Z 
2022-09-27T15:44:22.6199007Z Generating XML reports...
2022-09-27T15:44:22.6245754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154420.xml
2022-09-27T15:44:23.9145226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:23.9145855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:23.9153982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:24.2527960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:24.2528877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:24.2538616Z 
2022-09-27T15:44:24.2539092Z Running tests...
2022-09-27T15:44:24.2539532Z ----------------------------------------------------------------------
2022-09-27T15:44:24.4915514Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13676
2022-09-27T15:44:24.4940794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13677
2022-09-27T15:44:24.4968004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13678
2022-09-27T15:44:25.6019111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:25.6019770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:25.6189794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:25.6190212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:25.6364713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:25.6365360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:25.6374307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:25.6374825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:25.6544724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:25.6545278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:25.6713126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:25.6713646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:26.0018858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:26.0019551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:26.0020026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:26.0021019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:26.0021575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:26.0022101Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:26.0124192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:26.1031274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:26.1031849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:26.4018005Z skip: Need at least 2 CUDA devices (2.148s)
2022-09-27T15:44:26.4018300Z 
2022-09-27T15:44:26.4018801Z ----------------------------------------------------------------------
2022-09-27T15:44:26.4019252Z Ran 1 test in 2.148s
2022-09-27T15:44:26.4019393Z 
2022-09-27T15:44:26.4019474Z OK (skipped=1)
2022-09-27T15:44:26.4019582Z 
2022-09-27T15:44:26.4019669Z Generating XML reports...
2022-09-27T15:44:26.4066668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154424.xml
2022-09-27T15:44:27.7399593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:27.7399987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:27.7407252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:27.7407609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:28.0779758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:28.0789510Z 
2022-09-27T15:44:28.0789626Z Running tests...
2022-09-27T15:44:28.0790053Z ----------------------------------------------------------------------
2022-09-27T15:44:28.0806667Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:44:28.0807120Z 
2022-09-27T15:44:28.0807327Z ----------------------------------------------------------------------
2022-09-27T15:44:28.0807572Z Ran 1 test in 0.002s
2022-09-27T15:44:28.0807675Z 
2022-09-27T15:44:28.0807748Z OK (skipped=1)
2022-09-27T15:44:28.0807859Z 
2022-09-27T15:44:28.0808158Z Generating XML reports...
2022-09-27T15:44:28.0837209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154428.xml
2022-09-27T15:44:29.2945531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:29.2945934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:29.2953747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:29.2954134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:29.6325426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:29.6334673Z 
2022-09-27T15:44:29.6334797Z Running tests...
2022-09-27T15:44:29.6335432Z ----------------------------------------------------------------------
2022-09-27T15:44:29.8692886Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13739
2022-09-27T15:44:29.8719628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13740
2022-09-27T15:44:29.8746622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13741
2022-09-27T15:44:30.9460118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:30.9460791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:30.9784966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:30.9785629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:30.9951998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:30.9952658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:30.9953438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:30.9954019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:31.0311614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:31.0312293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:31.0313106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:31.0313715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:31.3804691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:31.3805351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:31.3806012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:31.3806982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:31.3807516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:31.3808045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:31.3812111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:31.3812850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:31.3813556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:31.6797178Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:44:31.6797503Z 
2022-09-27T15:44:31.6797912Z ----------------------------------------------------------------------
2022-09-27T15:44:31.6798250Z Ran 1 test in 2.046s
2022-09-27T15:44:31.6798365Z 
2022-09-27T15:44:31.6798441Z OK (skipped=1)
2022-09-27T15:44:31.6798551Z 
2022-09-27T15:44:31.6798624Z Generating XML reports...
2022-09-27T15:44:31.6843750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154429.xml
2022-09-27T15:44:32.9635602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:32.9635996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:32.9643265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:32.9643864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:33.3015959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:33.3024814Z 
2022-09-27T15:44:33.3025167Z Running tests...
2022-09-27T15:44:33.3025821Z ----------------------------------------------------------------------
2022-09-27T15:44:33.5371694Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13792
2022-09-27T15:44:33.5399113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13793
2022-09-27T15:44:33.5425789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13794
2022-09-27T15:44:34.6355582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:34.6356263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:34.6371791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:34.6372433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:34.6610771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:34.6611392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:34.6706388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:34.6707016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:34.6723531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:34.6724166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:34.6946553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:34.6947186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:35.0225034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:35.0324171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:35.0324601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:35.0325218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:35.0325749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:35.0326275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:35.0431409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:35.0432168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:35.1338571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:35.4475068Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:44:35.4475378Z 
2022-09-27T15:44:35.4475872Z ----------------------------------------------------------------------
2022-09-27T15:44:35.4476137Z Ran 1 test in 2.145s
2022-09-27T15:44:35.4476252Z 
2022-09-27T15:44:35.4476325Z OK (skipped=1)
2022-09-27T15:44:35.4476434Z 
2022-09-27T15:44:35.4476518Z Generating XML reports...
2022-09-27T15:44:35.4521882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154433.xml
2022-09-27T15:44:36.7407038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:36.7407428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:36.7414647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:36.7415055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:37.0818887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:37.0828584Z 
2022-09-27T15:44:37.0828735Z Running tests...
2022-09-27T15:44:37.0829373Z ----------------------------------------------------------------------
2022-09-27T15:44:37.3202391Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13845
2022-09-27T15:44:37.3229392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13846
2022-09-27T15:44:37.3256695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13847
2022-09-27T15:44:38.3810724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:38.3811095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:38.3971471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:38.3971836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:38.4070703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:38.4071210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:38.4155596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:38.4156097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:38.4327601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:38.4327965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:38.4411688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:38.4412180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:38.7668310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:38.7769143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:38.7769884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:38.7770602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:38.7771319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:38.7771836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:38.7875259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:38.7939269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6ue5fg9
2022-09-27T15:44:38.7941259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6ue5fg9/_remote_module_non_scriptable.py
2022-09-27T15:44:38.8780812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:38.8781463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:38.8882005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95x6taq3
2022-09-27T15:44:38.8882709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb7kwe6g
2022-09-27T15:44:38.8884509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95x6taq3/_remote_module_non_scriptable.py
2022-09-27T15:44:38.8885185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb7kwe6g/_remote_module_non_scriptable.py
2022-09-27T15:44:38.9076932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:44:38.9077669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:44:38.9078338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:44:39.3307765Z ok (2.247s)
2022-09-27T15:44:39.3308012Z 
2022-09-27T15:44:39.3308493Z ----------------------------------------------------------------------
2022-09-27T15:44:39.3308930Z Ran 1 test in 2.248s
2022-09-27T15:44:39.3309113Z 
2022-09-27T15:44:39.3309177Z OK
2022-09-27T15:44:39.3309268Z 
2022-09-27T15:44:39.3309376Z Generating XML reports...
2022-09-27T15:44:39.3353375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154437.xml
2022-09-27T15:44:40.6398907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:40.6399285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:40.6407018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:40.6407376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:40.9927185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:40.9936413Z 
2022-09-27T15:44:40.9936509Z Running tests...
2022-09-27T15:44:40.9937480Z ----------------------------------------------------------------------
2022-09-27T15:44:41.2343376Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13907
2022-09-27T15:44:41.2370633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13908
2022-09-27T15:44:41.2398262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13909
2022-09-27T15:44:42.2896916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:42.2897300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:42.3175738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:42.3176199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:42.3213118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:42.3213865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:42.3376282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:42.3376846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:42.3514781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:42.3515298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:42.3712516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:42.3712998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:42.7080512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:42.7181410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:42.7182108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:42.7183054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:42.7183595Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:42.7184221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:42.7189662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:42.7191098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:42.7191994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:43.0446537Z skip: CUDA is not available. (2.051s)
2022-09-27T15:44:43.0446904Z 
2022-09-27T15:44:43.0447386Z ----------------------------------------------------------------------
2022-09-27T15:44:43.0447650Z Ran 1 test in 2.051s
2022-09-27T15:44:43.0447774Z 
2022-09-27T15:44:43.0447847Z OK (skipped=1)
2022-09-27T15:44:43.0447955Z 
2022-09-27T15:44:43.0448027Z Generating XML reports...
2022-09-27T15:44:43.0494964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154440.xml
2022-09-27T15:44:44.3565670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:44.3566034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:44.3573395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:44.3573769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:44.6980286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:44.6990727Z 
2022-09-27T15:44:44.6991122Z Running tests...
2022-09-27T15:44:44.6991504Z ----------------------------------------------------------------------
2022-09-27T15:44:44.7011917Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T15:44:44.7012202Z 
2022-09-27T15:44:44.7012414Z ----------------------------------------------------------------------
2022-09-27T15:44:44.7012659Z Ran 1 test in 0.002s
2022-09-27T15:44:44.7012760Z 
2022-09-27T15:44:44.7012832Z OK (skipped=1)
2022-09-27T15:44:44.7012940Z 
2022-09-27T15:44:44.7013028Z Generating XML reports...
2022-09-27T15:44:44.7042727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154444.xml
2022-09-27T15:44:45.8949688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:45.8950080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:45.8957381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:45.8957926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:46.2457953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:46.2467391Z 
2022-09-27T15:44:46.2467497Z Running tests...
2022-09-27T15:44:46.2467908Z ----------------------------------------------------------------------
2022-09-27T15:44:46.2487789Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T15:44:46.2488263Z 
2022-09-27T15:44:46.2488635Z ----------------------------------------------------------------------
2022-09-27T15:44:46.2489099Z Ran 1 test in 0.002s
2022-09-27T15:44:46.2489306Z 
2022-09-27T15:44:46.2489421Z OK (skipped=1)
2022-09-27T15:44:46.2489621Z 
2022-09-27T15:44:46.2489786Z Generating XML reports...
2022-09-27T15:44:46.2518993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154446.xml
2022-09-27T15:44:47.4476974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:47.4477358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:47.4485336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:47.4485705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:47.7854226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:47.7863778Z 
2022-09-27T15:44:47.7863982Z Running tests...
2022-09-27T15:44:47.7864400Z ----------------------------------------------------------------------
2022-09-27T15:44:47.7881252Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:44:47.7881777Z 
2022-09-27T15:44:47.7881989Z ----------------------------------------------------------------------
2022-09-27T15:44:47.7882235Z Ran 1 test in 0.002s
2022-09-27T15:44:47.7882351Z 
2022-09-27T15:44:47.7882411Z OK (skipped=1)
2022-09-27T15:44:47.7882522Z 
2022-09-27T15:44:47.7882607Z Generating XML reports...
2022-09-27T15:44:47.7911646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154447.xml
2022-09-27T15:44:48.9994102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:48.9994770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:49.0002983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:49.0003616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:49.3446783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:49.3457352Z 
2022-09-27T15:44:49.3457729Z Running tests...
2022-09-27T15:44:49.3458138Z ----------------------------------------------------------------------
2022-09-27T15:44:49.3474478Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:44:49.3474872Z 
2022-09-27T15:44:49.3475130Z ----------------------------------------------------------------------
2022-09-27T15:44:49.3475490Z Ran 1 test in 0.002s
2022-09-27T15:44:49.3475615Z 
2022-09-27T15:44:49.3475689Z OK (skipped=1)
2022-09-27T15:44:49.3475853Z 
2022-09-27T15:44:49.3476450Z Generating XML reports...
2022-09-27T15:44:49.3506816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154449.xml
2022-09-27T15:44:50.5393007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:50.5393385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:50.5401817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:50.5402183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:50.8815943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:50.8825657Z 
2022-09-27T15:44:50.8825865Z Running tests...
2022-09-27T15:44:50.8826189Z ----------------------------------------------------------------------
2022-09-27T15:44:50.8850943Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T15:44:50.8851411Z 
2022-09-27T15:44:50.8851741Z ----------------------------------------------------------------------
2022-09-27T15:44:50.8851990Z Ran 1 test in 0.002s
2022-09-27T15:44:50.8852092Z 
2022-09-27T15:44:50.8852164Z OK (skipped=1)
2022-09-27T15:44:50.8852272Z 
2022-09-27T15:44:50.8852359Z Generating XML reports...
2022-09-27T15:44:50.8883279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154450.xml
2022-09-27T15:44:52.0828243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:52.0828623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:52.0836033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:52.0836411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:52.4219588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:52.4229808Z 
2022-09-27T15:44:52.4230125Z Running tests...
2022-09-27T15:44:52.4230739Z ----------------------------------------------------------------------
2022-09-27T15:44:52.6600711Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14010
2022-09-27T15:44:52.6628508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14011
2022-09-27T15:44:52.6655567Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14012
2022-09-27T15:44:53.7653878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:53.7654268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:53.7663168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:53.7663599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:53.8007217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:53.8007657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:53.8012685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:53.8013060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:53.8019443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:53.8020077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:53.8362222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:53.8362823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:54.1908257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:54.1909030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:54.1909458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:54.1910081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:54.1910619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:54.1911166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:54.1916534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:54.1917133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:54.1917733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:54.4702830Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T15:44:54.4703154Z 
2022-09-27T15:44:54.4703584Z ----------------------------------------------------------------------
2022-09-27T15:44:54.4703973Z Ran 1 test in 2.047s
2022-09-27T15:44:54.4704098Z 
2022-09-27T15:44:54.4704160Z OK (skipped=1)
2022-09-27T15:44:54.4704273Z 
2022-09-27T15:44:54.4704362Z Generating XML reports...
2022-09-27T15:44:54.4748157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154452.xml
2022-09-27T15:44:55.7916250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:55.7916645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:55.7923948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:55.7924312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:56.1278700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:56.1287590Z 
2022-09-27T15:44:56.1287726Z Running tests...
2022-09-27T15:44:56.1288165Z ----------------------------------------------------------------------
2022-09-27T15:44:56.3610285Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14063
2022-09-27T15:44:56.3637728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14064
2022-09-27T15:44:56.3664880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14065
2022-09-27T15:44:57.4555178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:57.4555587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:57.4881560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:57.4951063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:57.4951855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:57.4952192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:57.4984796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:57.4985440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:57.5302783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:57.5303650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:57.5334832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:57.5335427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:57.8766578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:44:57.8767249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:44:57.8767646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:44:57.8768255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:57.8769020Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:57.8769878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:44:57.8775451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:44:57.8775861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:44:57.9779656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:44:58.2713235Z skip: Need at least 2 CUDA devices (2.142s)
2022-09-27T15:44:58.2713474Z 
2022-09-27T15:44:58.2713824Z ----------------------------------------------------------------------
2022-09-27T15:44:58.2714061Z Ran 1 test in 2.142s
2022-09-27T15:44:58.2714177Z 
2022-09-27T15:44:58.2714267Z OK (skipped=1)
2022-09-27T15:44:58.2714376Z 
2022-09-27T15:44:58.2714463Z Generating XML reports...
2022-09-27T15:44:58.2759756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154456.xml
2022-09-27T15:44:59.5742297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:44:59.5742679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:44:59.5751196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:44:59.5751691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:44:59.9121340Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:44:59.9130738Z 
2022-09-27T15:44:59.9130877Z Running tests...
2022-09-27T15:44:59.9131495Z ----------------------------------------------------------------------
2022-09-27T15:44:59.9148134Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:44:59.9148509Z 
2022-09-27T15:44:59.9148710Z ----------------------------------------------------------------------
2022-09-27T15:44:59.9148961Z Ran 1 test in 0.002s
2022-09-27T15:44:59.9149081Z 
2022-09-27T15:44:59.9149199Z OK (skipped=1)
2022-09-27T15:44:59.9149330Z 
2022-09-27T15:44:59.9149417Z Generating XML reports...
2022-09-27T15:44:59.9178394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154459.xml
2022-09-27T15:45:01.1104931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:01.1105364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:01.1113543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:01.1113914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:01.4530937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:01.4540005Z 
2022-09-27T15:45:01.4540125Z Running tests...
2022-09-27T15:45:01.4540676Z ----------------------------------------------------------------------
2022-09-27T15:45:01.4558916Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:45:01.4559198Z 
2022-09-27T15:45:01.4559413Z ----------------------------------------------------------------------
2022-09-27T15:45:01.4559658Z Ran 1 test in 0.002s
2022-09-27T15:45:01.4559760Z 
2022-09-27T15:45:01.4559833Z OK (skipped=1)
2022-09-27T15:45:01.4559941Z 
2022-09-27T15:45:01.4560027Z Generating XML reports...
2022-09-27T15:45:01.4589370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154501.xml
2022-09-27T15:45:02.6572761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:02.6573174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:02.6580185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:02.6580541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:02.9977545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:02.9987076Z 
2022-09-27T15:45:02.9987297Z Running tests...
2022-09-27T15:45:02.9987648Z ----------------------------------------------------------------------
2022-09-27T15:45:03.2360528Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14136
2022-09-27T15:45:03.2387164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14137
2022-09-27T15:45:03.2414239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14138
2022-09-27T15:45:04.3181960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:04.3182596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:04.3213817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:04.3214469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:04.3525090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:04.3525825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:04.3561265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:04.3561866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:04.3860160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:04.3860750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:04.4177909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:04.4178312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:04.7531224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:04.7631454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:04.7632279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:04.7632931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:04.7633539Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:04.7634050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:04.7738954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:04.7739738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:04.8644655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:05.1462279Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:45:05.1462538Z 
2022-09-27T15:45:05.1463053Z ----------------------------------------------------------------------
2022-09-27T15:45:05.1463525Z Ran 1 test in 2.147s
2022-09-27T15:45:05.1463731Z 
2022-09-27T15:45:05.1463815Z OK (skipped=1)
2022-09-27T15:45:05.1464106Z 
2022-09-27T15:45:05.1464212Z Generating XML reports...
2022-09-27T15:45:05.1509504Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154502.xml
2022-09-27T15:45:06.4461778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:06.4462156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:06.4470494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:06.4470866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:06.7834017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:06.7843272Z 
2022-09-27T15:45:06.7843573Z Running tests...
2022-09-27T15:45:07.0208466Z ----------------------------------------------------------------------
2022-09-27T15:45:07.0209285Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14189
2022-09-27T15:45:07.0235459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14190
2022-09-27T15:45:07.0262309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14191
2022-09-27T15:45:08.1162365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:08.1163024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:08.1494159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:08.1494791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:08.1550396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:08.1551077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:08.1876700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:08.1877317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:08.1878019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:08.1878550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:08.2204995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:08.2205637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:08.5485736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:08.5486217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:08.5486731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:08.5487422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:08.5487981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:08.5488596Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:08.5591587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:08.6499138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:08.6499708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:08.9314619Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:45:08.9314915Z 
2022-09-27T15:45:08.9315362Z ----------------------------------------------------------------------
2022-09-27T15:45:08.9315734Z Ran 1 test in 2.147s
2022-09-27T15:45:08.9315917Z 
2022-09-27T15:45:08.9316035Z OK (skipped=1)
2022-09-27T15:45:08.9316209Z 
2022-09-27T15:45:08.9316343Z Generating XML reports...
2022-09-27T15:45:08.9362633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154506.xml
2022-09-27T15:45:10.2331184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:10.2331561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:10.2339269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:10.2339633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:10.5713025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:10.5722660Z 
2022-09-27T15:45:10.5722763Z Running tests...
2022-09-27T15:45:10.5723222Z ----------------------------------------------------------------------
2022-09-27T15:45:10.5747478Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:45:10.5747909Z 
2022-09-27T15:45:10.5748192Z ----------------------------------------------------------------------
2022-09-27T15:45:10.5748467Z Ran 1 test in 0.002s
2022-09-27T15:45:10.5748585Z 
2022-09-27T15:45:10.5748659Z OK (skipped=1)
2022-09-27T15:45:10.5748766Z 
2022-09-27T15:45:10.5748839Z Generating XML reports...
2022-09-27T15:45:10.5778513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154510.xml
2022-09-27T15:45:11.7944473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:11.7944860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:11.7953247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:11.7953615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:12.1347556Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:12.1356422Z 
2022-09-27T15:45:12.1356518Z Running tests...
2022-09-27T15:45:12.1357082Z ----------------------------------------------------------------------
2022-09-27T15:45:12.3732153Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14252
2022-09-27T15:45:12.3757728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14253
2022-09-27T15:45:12.3785710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14254
2022-09-27T15:45:13.4802968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:13.4803540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:13.4886108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:13.4886677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:13.5140916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:13.5141580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:13.5233393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:13.5234004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:13.5360942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:13.5361601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:13.5679621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:13.5680253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:13.9122106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:13.9122865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:13.9123277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:13.9123908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:13.9124459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:13.9124969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:13.9228279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:14.0133874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:14.0134402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:14.2833306Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:45:14.2833707Z 
2022-09-27T15:45:14.2834135Z ----------------------------------------------------------------------
2022-09-27T15:45:14.2834410Z Ran 1 test in 2.148s
2022-09-27T15:45:14.2834542Z 
2022-09-27T15:45:14.2834614Z OK (skipped=1)
2022-09-27T15:45:14.2834710Z 
2022-09-27T15:45:14.2834798Z Generating XML reports...
2022-09-27T15:45:14.2879546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154512.xml
2022-09-27T15:45:15.5682943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:15.5683365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:15.5690629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:15.5691198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:15.9063138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:15.9072473Z 
2022-09-27T15:45:15.9072614Z Running tests...
2022-09-27T15:45:15.9073074Z ----------------------------------------------------------------------
2022-09-27T15:45:16.1439834Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14305
2022-09-27T15:45:16.1467249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14306
2022-09-27T15:45:16.1494675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14307
2022-09-27T15:45:17.2220715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:17.2221372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:17.2530403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:17.2531094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:17.2876863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:17.2877253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:17.2879726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:17.2880232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:17.3229091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:17.3229706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:17.3231709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:17.3232332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:17.6698033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:17.6799113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:17.6801764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:17.6802248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:17.6802778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:17.6803313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:17.6807564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:17.6809139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:17.6810683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:17.9540611Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:45:17.9540874Z 
2022-09-27T15:45:17.9541263Z ----------------------------------------------------------------------
2022-09-27T15:45:17.9541518Z Ran 1 test in 2.047s
2022-09-27T15:45:17.9541638Z 
2022-09-27T15:45:17.9541711Z OK (skipped=1)
2022-09-27T15:45:17.9541820Z 
2022-09-27T15:45:17.9541904Z Generating XML reports...
2022-09-27T15:45:17.9586537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154515.xml
2022-09-27T15:45:19.2532513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:19.2533119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:19.2539533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:19.2540124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:19.5923571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:19.5933498Z 
2022-09-27T15:45:19.5933935Z Running tests...
2022-09-27T15:45:19.5934332Z ----------------------------------------------------------------------
2022-09-27T15:45:19.8307350Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14358
2022-09-27T15:45:19.8334113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14359
2022-09-27T15:45:19.8361534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14360
2022-09-27T15:45:20.9325273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:20.9325847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:20.9486478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:20.9487159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:20.9663536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:20.9664300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:20.9732341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:20.9732886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:20.9838205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:20.9838765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:21.0067642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:21.0068290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:21.3465081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:21.3531871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:21.3532248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:21.3532878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:21.3533423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:21.3566643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:21.3639548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:21.3640414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:21.4578800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:21.7409746Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:45:21.7410014Z 
2022-09-27T15:45:21.7410523Z ----------------------------------------------------------------------
2022-09-27T15:45:21.7410979Z Ran 1 test in 2.148s
2022-09-27T15:45:21.7411180Z 
2022-09-27T15:45:21.7411313Z OK (skipped=1)
2022-09-27T15:45:21.7411491Z 
2022-09-27T15:45:21.7411644Z Generating XML reports...
2022-09-27T15:45:21.7458976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154519.xml
2022-09-27T15:45:23.0676872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:23.0677512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:23.0684635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:23.0684993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:23.4196765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:23.4206910Z 
2022-09-27T15:45:23.4207088Z Running tests...
2022-09-27T15:45:23.4207500Z ----------------------------------------------------------------------
2022-09-27T15:45:23.6565891Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14411
2022-09-27T15:45:23.6591446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14412
2022-09-27T15:45:23.6618660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14413
2022-09-27T15:45:24.7174868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:24.7175500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:24.7388753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:24.7389454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:24.7502050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:24.7502589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:24.7715744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:24.7716449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:24.8010234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:24.8010844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:24.8325357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:24.8326066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:25.1708957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:25.1809114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:25.1809807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:25.1810615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:25.1811142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:25.1811666Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:25.1915435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:25.2821526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:25.2823067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:25.5667023Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T15:45:25.5667298Z 
2022-09-27T15:45:25.5667996Z ----------------------------------------------------------------------
2022-09-27T15:45:25.5668466Z Ran 1 test in 2.146s
2022-09-27T15:45:25.5668615Z 
2022-09-27T15:45:25.5668701Z OK (skipped=1)
2022-09-27T15:45:25.5668880Z 
2022-09-27T15:45:25.5668971Z Generating XML reports...
2022-09-27T15:45:25.5714046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154523.xml
2022-09-27T15:45:26.8874631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:26.8875029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:26.8882706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:26.8883249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:27.2252431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:27.2261504Z 
2022-09-27T15:45:27.2261637Z Running tests...
2022-09-27T15:45:27.2262246Z ----------------------------------------------------------------------
2022-09-27T15:45:27.4614829Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14464
2022-09-27T15:45:27.4641923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14465
2022-09-27T15:45:27.4669235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14466
2022-09-27T15:45:28.5301840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:28.5302489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:28.5623644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:28.5624421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:28.5635123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:28.5635680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:28.5952512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:28.5952910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:28.6235781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:28.6236308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:28.6551590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:28.6552025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:28.9808648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:28.9809075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:28.9809471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:28.9810081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:28.9810618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:28.9811148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:28.9914238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:29.0820857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:29.0821323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:29.3718966Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:45:29.3719255Z 
2022-09-27T15:45:29.3719781Z ----------------------------------------------------------------------
2022-09-27T15:45:29.3720177Z Ran 1 test in 2.146s
2022-09-27T15:45:29.3720290Z 
2022-09-27T15:45:29.3720351Z OK (skipped=1)
2022-09-27T15:45:29.3720464Z 
2022-09-27T15:45:29.3720549Z Generating XML reports...
2022-09-27T15:45:29.3766203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154527.xml
2022-09-27T15:45:30.6808771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:30.6809218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:30.6816811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:30.6817184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:31.0210522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:31.0220467Z 
2022-09-27T15:45:31.0220601Z Running tests...
2022-09-27T15:45:31.0221048Z ----------------------------------------------------------------------
2022-09-27T15:45:31.2562419Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-09-27T15:45:31.2562918Z 
2022-09-27T15:45:31.2563157Z ----------------------------------------------------------------------
2022-09-27T15:45:31.2563406Z Ran 1 test in 0.234s
2022-09-27T15:45:31.2563509Z 
2022-09-27T15:45:31.2563583Z OK (skipped=1)
2022-09-27T15:45:31.2563694Z 
2022-09-27T15:45:31.2563789Z Generating XML reports...
2022-09-27T15:45:31.2587569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154531.xml
2022-09-27T15:45:32.5276486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:32.5276893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:32.5284941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:32.5285303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:32.8874044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:32.8883466Z 
2022-09-27T15:45:32.8883568Z Running tests...
2022-09-27T15:45:32.8884115Z ----------------------------------------------------------------------
2022-09-27T15:45:33.1241662Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-09-27T15:45:33.1242183Z 
2022-09-27T15:45:33.1242410Z ----------------------------------------------------------------------
2022-09-27T15:45:33.1242656Z Ran 1 test in 0.236s
2022-09-27T15:45:33.1242757Z 
2022-09-27T15:45:33.1242831Z OK (skipped=1)
2022-09-27T15:45:33.1242942Z 
2022-09-27T15:45:33.1243027Z Generating XML reports...
2022-09-27T15:45:33.1265210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154532.xml
2022-09-27T15:45:34.3962976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:34.3963622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:34.3971532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:34.3971933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:34.7372164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:34.7381733Z 
2022-09-27T15:45:34.7381831Z Running tests...
2022-09-27T15:45:34.7382273Z ----------------------------------------------------------------------
2022-09-27T15:45:34.7409735Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-09-27T15:45:34.7410199Z 
2022-09-27T15:45:34.7410516Z ----------------------------------------------------------------------
2022-09-27T15:45:34.7410795Z Ran 1 test in 0.003s
2022-09-27T15:45:34.7410918Z 
2022-09-27T15:45:34.7410994Z OK (skipped=1)
2022-09-27T15:45:34.7411103Z 
2022-09-27T15:45:34.7411189Z Generating XML reports...
2022-09-27T15:45:34.7441028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154534.xml
2022-09-27T15:45:35.9325403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:35.9326069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:35.9333750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:35.9334518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:36.2717735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:36.2727249Z 
2022-09-27T15:45:36.2727706Z Running tests...
2022-09-27T15:45:36.2728122Z ----------------------------------------------------------------------
2022-09-27T15:45:36.5067545Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14547
2022-09-27T15:45:36.5094221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14548
2022-09-27T15:45:36.5121164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14549
2022-09-27T15:45:37.5796088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:37.5796761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:37.5925856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:37.5926537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:37.5940032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:37.5940696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:37.6141214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:37.6141622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:37.6279320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:37.6279926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:37.6298312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:37.6299171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:37.9764435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:37.9765326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:37.9765708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:37.9766354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:37.9766954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:37.9767468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:37.9870476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:38.0773926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:38.0774461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:38.0980316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:45:38.1082012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:45:38.1082587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:45:38.1083540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:38.1084266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:38.1084807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:38.4172351Z ok (2.144s)
2022-09-27T15:45:38.4172592Z 
2022-09-27T15:45:38.4173007Z ----------------------------------------------------------------------
2022-09-27T15:45:38.4173369Z Ran 1 test in 2.144s
2022-09-27T15:45:38.4173486Z 
2022-09-27T15:45:38.4173549Z OK
2022-09-27T15:45:38.4174945Z 
2022-09-27T15:45:38.4175163Z Generating XML reports...
2022-09-27T15:45:38.4219227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154536.xml
2022-09-27T15:45:39.7457304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:39.7457738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:39.7465115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:39.7465568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:40.0865871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:40.0875427Z 
2022-09-27T15:45:40.0875537Z Running tests...
2022-09-27T15:45:40.0876080Z ----------------------------------------------------------------------
2022-09-27T15:45:40.3237148Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14609
2022-09-27T15:45:40.3262824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14610
2022-09-27T15:45:40.3290145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14611
2022-09-27T15:45:41.4165180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:41.4165567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:41.4308427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:41.4308827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:41.4400379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:41.4400734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:41.4508418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:41.4508902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:41.4669062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:41.4669597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:41.4742571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:41.4742941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:41.8094873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:41.8195458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:41.8195880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:41.8196493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:41.8197018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:41.8197542Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:41.8302560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:41.9207637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:41.9208075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:41.9208449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:45:41.9312850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:45:41.9313860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:41.9314319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:45:41.9314841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:41.9411879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:45:42.3340147Z ok (2.246s)
2022-09-27T15:45:42.3340448Z 
2022-09-27T15:45:42.3340969Z ----------------------------------------------------------------------
2022-09-27T15:45:42.3341289Z Ran 1 test in 2.246s
2022-09-27T15:45:42.3341409Z 
2022-09-27T15:45:42.3341471Z OK
2022-09-27T15:45:42.3341564Z 
2022-09-27T15:45:42.3341659Z Generating XML reports...
2022-09-27T15:45:42.3386168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154540.xml
2022-09-27T15:45:43.6343209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:43.6343599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:43.6351964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:43.6352322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:43.9722579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:43.9731803Z 
2022-09-27T15:45:43.9731994Z Running tests...
2022-09-27T15:45:43.9732342Z ----------------------------------------------------------------------
2022-09-27T15:45:44.2094995Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14668
2022-09-27T15:45:44.2120675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14669
2022-09-27T15:45:44.2148042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14670
2022-09-27T15:45:45.2478455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:45.2799551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:45.2800141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:45.2800525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:45.2971529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:45.2971877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:45.3269450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:45.3269877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:45.3286581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:45.3287008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:45.3597247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:45.3597843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:45.6851506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:45.6952238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:45.6952912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:45.6953854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:45.6954414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:45.6955118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:45.6961266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:45.6962115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:45.6962755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:46.0196894Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:45:46.0197093Z 
2022-09-27T15:45:46.0197399Z ----------------------------------------------------------------------
2022-09-27T15:45:46.0197649Z Ran 1 test in 2.046s
2022-09-27T15:45:46.0197752Z 
2022-09-27T15:45:46.0197829Z OK (skipped=1)
2022-09-27T15:45:46.0197939Z 
2022-09-27T15:45:46.0198025Z Generating XML reports...
2022-09-27T15:45:46.0242135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154543.xml
2022-09-27T15:45:47.3019491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:47.3019932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:47.3027363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:47.3027734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:47.6385554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:47.6395352Z 
2022-09-27T15:45:47.6395774Z Running tests...
2022-09-27T15:45:47.6396405Z ----------------------------------------------------------------------
2022-09-27T15:45:47.6414045Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T15:45:47.6414373Z 
2022-09-27T15:45:47.6414702Z ----------------------------------------------------------------------
2022-09-27T15:45:47.6415177Z Ran 1 test in 0.002s
2022-09-27T15:45:47.6415375Z 
2022-09-27T15:45:47.6415505Z OK (skipped=1)
2022-09-27T15:45:47.6415625Z 
2022-09-27T15:45:47.6415720Z Generating XML reports...
2022-09-27T15:45:47.6444916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154547.xml
2022-09-27T15:45:48.8286665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:48.8287035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:48.8295278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:48.8295661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:49.1668567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:49.1678065Z 
2022-09-27T15:45:49.1678574Z Running tests...
2022-09-27T15:45:49.1679156Z ----------------------------------------------------------------------
2022-09-27T15:45:49.4026040Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14731
2022-09-27T15:45:49.4052783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14732
2022-09-27T15:45:49.4080116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14733
2022-09-27T15:45:50.4585624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:50.4586294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:50.4899837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:50.4900540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:50.4904537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:50.4905244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:50.5182113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:50.5182483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:50.5231588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:50.5232186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:50.5505186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:50.5505837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:50.8863650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:50.8964892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:50.8966067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:50.8966483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:50.8966972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:50.8967506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:50.8972289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:50.8973632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:50.8974897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:51.2130515Z ok (2.045s)
2022-09-27T15:45:51.2130798Z 
2022-09-27T15:45:51.2131263Z ----------------------------------------------------------------------
2022-09-27T15:45:51.2131523Z Ran 1 test in 2.045s
2022-09-27T15:45:51.2131640Z 
2022-09-27T15:45:51.2131704Z OK
2022-09-27T15:45:51.2131797Z 
2022-09-27T15:45:51.2131892Z Generating XML reports...
2022-09-27T15:45:51.2176966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154549.xml
2022-09-27T15:45:52.4971162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:52.4971546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:52.4978316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:52.4978703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:52.8363232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:52.8372104Z 
2022-09-27T15:45:52.8372357Z Running tests...
2022-09-27T15:45:52.8372797Z ----------------------------------------------------------------------
2022-09-27T15:45:53.0761315Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14784
2022-09-27T15:45:53.0787837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14785
2022-09-27T15:45:53.0814788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14786
2022-09-27T15:45:54.1797531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:54.1798203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:54.1824479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:54.1824872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:54.2146380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:54.2147061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:54.2177114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:54.2177723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:54.2427259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:54.2427631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:54.2761516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:54.2761912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:54.6287331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:54.6388859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:54.6389910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:54.6390369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:54.6390907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:54.6391760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:54.6398380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:54.6398776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:54.6402791Z STAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.6403297Z STAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7398098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:54.7403410Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7409901Z STAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7410366Z 
2022-09-27T15:45:54.7410797Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7414675Z STAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7415118Z 
2022-09-27T15:45:54.7418535Z STAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7421972Z STAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7426909Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7430959Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7431579Z STAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7434283Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7437363Z STAGE:2022-09-27 15:45:54 14784:14784 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7444236Z STAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:45:54.7447161Z STAGE:2022-09-27 15:45:54 14785:14785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:54.7448204Z STAGE:2022-09-27 15:45:54 14786:14786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:45:55.1866774Z ok (2.349s)
2022-09-27T15:45:55.1867007Z 
2022-09-27T15:45:55.1867451Z ----------------------------------------------------------------------
2022-09-27T15:45:55.1867888Z Ran 1 test in 2.349s
2022-09-27T15:45:55.1868095Z 
2022-09-27T15:45:55.1868192Z OK
2022-09-27T15:45:55.1868292Z 
2022-09-27T15:45:55.1868576Z Generating XML reports...
2022-09-27T15:45:55.1912295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154552.xml
2022-09-27T15:45:56.5213506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:56.5214127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:56.5222497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:56.5223142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:56.8589860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:45:56.8599251Z 
2022-09-27T15:45:56.8599635Z Running tests...
2022-09-27T15:45:56.8600040Z ----------------------------------------------------------------------
2022-09-27T15:45:57.0951019Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14840
2022-09-27T15:45:57.0976710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14841
2022-09-27T15:45:57.1003334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14842
2022-09-27T15:45:58.2068191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:58.2068849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:58.2074416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:58.2075019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:58.2420205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:58.2420957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:58.2425722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:58.2426275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:58.2524862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:45:58.2525460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:45:58.2859503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:45:58.2860160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:45:58.6326665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:45:58.6427509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:45:58.6428628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:58.6429047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:45:58.6429558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:58.6430074Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:45:58.6434896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:45:58.6435483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:45:58.6437574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:45:59.0051771Z ok (2.145s)
2022-09-27T15:45:59.0052066Z 
2022-09-27T15:45:59.0052842Z ----------------------------------------------------------------------
2022-09-27T15:45:59.0053297Z Ran 1 test in 2.145s
2022-09-27T15:45:59.0053455Z 
2022-09-27T15:45:59.0053599Z OK
2022-09-27T15:45:59.0053683Z 
2022-09-27T15:45:59.0053781Z Generating XML reports...
2022-09-27T15:45:59.0098373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154556.xml
2022-09-27T15:46:00.2885473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:00.2885852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:00.2893351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:00.2893710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:00.6316492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:00.6326263Z 
2022-09-27T15:46:00.6326738Z Running tests...
2022-09-27T15:46:00.6327183Z ----------------------------------------------------------------------
2022-09-27T15:46:00.6342322Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-09-27T15:46:00.6342718Z 
2022-09-27T15:46:00.6343117Z ----------------------------------------------------------------------
2022-09-27T15:46:00.6343499Z Ran 1 test in 0.002s
2022-09-27T15:46:00.6343682Z 
2022-09-27T15:46:00.6343796Z OK (skipped=1)
2022-09-27T15:46:00.6344187Z 
2022-09-27T15:46:00.6344346Z Generating XML reports...
2022-09-27T15:46:00.6374711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154600.xml
2022-09-27T15:46:01.8596331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:01.8596969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:01.8605058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:01.8605744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:02.2015068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:02.2025117Z 
2022-09-27T15:46:02.2025526Z Running tests...
2022-09-27T15:46:02.2025947Z ----------------------------------------------------------------------
2022-09-27T15:46:02.4391196Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14903
2022-09-27T15:46:02.4416906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14904
2022-09-27T15:46:02.4444748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14905
2022-09-27T15:46:03.5169544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:03.5170149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:03.5240224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:03.5240947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:03.5295980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:03.5296629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:03.5539081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:03.5539511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:03.5606784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:03.5607164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:03.5656488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:03.5656852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:03.9008426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:03.9109523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:03.9110326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:03.9111568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:03.9112178Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:03.9112701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:03.9117334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:03.9118429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:03.9119977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:03.9326022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:46:03.9428256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:46:03.9428884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:46:03.9429746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:03.9430564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:03.9431446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:03.9433447Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9434188Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9434755Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9438550Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9439262Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9439954Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9443180Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9443740Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9447633Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9451160Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9454236Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9458178Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9459155Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9461281Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9464816Z STAGE:2022-09-27 15:46:03 14903:14903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9473094Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:03.9476046Z STAGE:2022-09-27 15:46:03 14904:14904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:03.9477131Z STAGE:2022-09-27 15:46:03 14905:14905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:04.4494393Z ok (2.247s)
2022-09-27T15:46:04.4494637Z 
2022-09-27T15:46:04.4495213Z ----------------------------------------------------------------------
2022-09-27T15:46:04.4495547Z Ran 1 test in 2.247s
2022-09-27T15:46:04.4495668Z 
2022-09-27T15:46:04.4495731Z OK
2022-09-27T15:46:04.4495826Z 
2022-09-27T15:46:04.4495924Z Generating XML reports...
2022-09-27T15:46:04.4541450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154602.xml
2022-09-27T15:46:05.7581317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:05.7581708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:05.7588945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:05.7589312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:06.0959079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:06.0969146Z 
2022-09-27T15:46:06.0969267Z Running tests...
2022-09-27T15:46:06.0969860Z ----------------------------------------------------------------------
2022-09-27T15:46:06.3312753Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14968
2022-09-27T15:46:06.3338839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14969
2022-09-27T15:46:06.3368132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14970
2022-09-27T15:46:07.3849445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:07.3850139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:07.4103120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:07.4103759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:07.4174755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:07.4175394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:07.4181695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:07.4182313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:07.4454619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:07.4455267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:07.4524925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:07.4525567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:07.8060849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:07.8061486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:07.8062117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:07.8062780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:07.8063481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:07.8064557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:07.8068715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:07.8069346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:07.8070302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:46:07.8070954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:07.8276382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:46:07.8276828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:46:07.8277417Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:07.8277991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:07.8280468Z STAGE:2022-09-27 15:46:07 14969:14969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:07.8374538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:07.8380612Z STAGE:2022-09-27 15:46:07 14970:14970 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:07.8385069Z STAGE:2022-09-27 15:46:07 14970:14970 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:07.8386277Z STAGE:2022-09-27 15:46:07 14969:14969 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:07.8389008Z STAGE:2022-09-27 15:46:07 14970:14970 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:07.8398611Z STAGE:2022-09-27 15:46:07 14969:14969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:07.8402530Z STAGE:2022-09-27 15:46:07 14969:14969 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:07.8402985Z STAGE:2022-09-27 15:46:07 14970:14970 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:08.2417762Z ok (2.144s)
2022-09-27T15:46:08.2417985Z 
2022-09-27T15:46:08.2418330Z ----------------------------------------------------------------------
2022-09-27T15:46:08.2418671Z Ran 1 test in 2.145s
2022-09-27T15:46:08.2418865Z 
2022-09-27T15:46:08.2418938Z OK
2022-09-27T15:46:08.2420224Z 
2022-09-27T15:46:08.2420544Z Generating XML reports...
2022-09-27T15:46:08.2466160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154606.xml
2022-09-27T15:46:09.5497121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:09.5497495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:09.5505228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:09.5505674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:09.8902430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:09.8911466Z 
2022-09-27T15:46:09.8911604Z Running tests...
2022-09-27T15:46:09.8912531Z ----------------------------------------------------------------------
2022-09-27T15:46:10.1254200Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15029
2022-09-27T15:46:10.1281658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15030
2022-09-27T15:46:10.1309089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15031
2022-09-27T15:46:11.2213098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:11.2213765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:11.2217160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:11.2217538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:11.2358387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:11.2359045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:11.2575413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:11.2576181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:11.2581129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:11.2581704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:11.2724850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:11.2725521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:11.6160913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:11.6261844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:11.6262471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:11.6263454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:11.6264405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:11.6265366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:11.6368168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:11.7272914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:11.7273545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:12.0358661Z ok (2.144s)
2022-09-27T15:46:12.0358894Z 
2022-09-27T15:46:12.0359352Z ----------------------------------------------------------------------
2022-09-27T15:46:12.0359771Z Ran 1 test in 2.145s
2022-09-27T15:46:12.0359928Z 
2022-09-27T15:46:12.0360018Z OK
2022-09-27T15:46:12.0360161Z 
2022-09-27T15:46:12.0360308Z Generating XML reports...
2022-09-27T15:46:12.0405466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154609.xml
2022-09-27T15:46:13.3428747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:13.3429139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:13.3436972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:13.3437691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:13.6817704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:13.6827257Z 
2022-09-27T15:46:13.6827714Z Running tests...
2022-09-27T15:46:13.6828301Z ----------------------------------------------------------------------
2022-09-27T15:46:13.9124193Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.229s)
2022-09-27T15:46:13.9124693Z 
2022-09-27T15:46:13.9124915Z ----------------------------------------------------------------------
2022-09-27T15:46:13.9125163Z Ran 1 test in 0.229s
2022-09-27T15:46:13.9125277Z 
2022-09-27T15:46:13.9125350Z OK (skipped=1)
2022-09-27T15:46:13.9125449Z 
2022-09-27T15:46:13.9125551Z Generating XML reports...
2022-09-27T15:46:13.9146298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154613.xml
2022-09-27T15:46:15.1621586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:15.1621975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:15.1630007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:15.1630518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:15.5031905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:15.5053824Z 
2022-09-27T15:46:15.5054354Z Running tests...
2022-09-27T15:46:15.5054768Z ----------------------------------------------------------------------
2022-09-27T15:46:15.7405565Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15092
2022-09-27T15:46:15.7431156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15093
2022-09-27T15:46:15.7457611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15094
2022-09-27T15:46:16.7829915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:16.7830300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:16.8146967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:16.8147570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:16.8328452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:16.8328850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:16.8642170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:16.8642689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:16.8767484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:16.8768017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:16.9090778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:16.9098605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:17.2553791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:17.2654662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:17.2655439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:17.2656222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:17.2657202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:17.2657926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:17.2760552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:17.3665048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:17.3665971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:17.3667138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:46:17.3771261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:46:17.3772084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:46:17.3772963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:17.3773854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:17.3871272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:17.7508104Z ok (2.246s)
2022-09-27T15:46:17.7508341Z 
2022-09-27T15:46:17.7508885Z ----------------------------------------------------------------------
2022-09-27T15:46:17.7509236Z Ran 1 test in 2.246s
2022-09-27T15:46:17.7509352Z 
2022-09-27T15:46:17.7509414Z OK
2022-09-27T15:46:17.7509505Z 
2022-09-27T15:46:17.7509597Z Generating XML reports...
2022-09-27T15:46:17.7553507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154615.xml
2022-09-27T15:46:19.0651715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:19.0652115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:19.0659636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:19.0659998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:19.4043031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:19.4053211Z 
2022-09-27T15:46:19.4053499Z Running tests...
2022-09-27T15:46:19.4053901Z ----------------------------------------------------------------------
2022-09-27T15:46:19.6400223Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15151
2022-09-27T15:46:19.6427696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15152
2022-09-27T15:46:19.6454125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15153
2022-09-27T15:46:20.7510425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:20.7511087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:20.7638123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:20.7654651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:20.7725559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:20.7726070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:20.7862629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:20.7863529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:20.7999744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:20.8000405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:20.8082122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:20.8082796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:21.1361898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:21.1362623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:21.1363246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:21.1363668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:21.1364156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:21.1364677Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:21.1467059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:21.2371913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:21.2372594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:21.5506417Z ok (2.145s)
2022-09-27T15:46:21.5506670Z 
2022-09-27T15:46:21.5507188Z ----------------------------------------------------------------------
2022-09-27T15:46:21.5507643Z Ran 1 test in 2.145s
2022-09-27T15:46:21.5507834Z 
2022-09-27T15:46:21.5507949Z OK
2022-09-27T15:46:21.5508116Z 
2022-09-27T15:46:21.5508279Z Generating XML reports...
2022-09-27T15:46:21.5554101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154619.xml
2022-09-27T15:46:22.8783234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:22.8783610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:22.8791423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:22.8791884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:23.2208768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:23.2217946Z 
2022-09-27T15:46:23.2218366Z Running tests...
2022-09-27T15:46:23.2218792Z ----------------------------------------------------------------------
2022-09-27T15:46:23.4643805Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15204
2022-09-27T15:46:23.4670739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15205
2022-09-27T15:46:23.4698522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15206
2022-09-27T15:46:24.4949632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:24.4950302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:24.5142080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:24.5142660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:24.5287167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:24.5287832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:24.5454428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:24.5455108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:24.5482149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:24.5482779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:24.5777470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:24.5778126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:24.8936332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:24.9037768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:24.9038246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:24.9039210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:24.9039971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:24.9040888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:24.9143308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:25.0047363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:25.0047802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:25.4748915Z ok (2.253s)
2022-09-27T15:46:25.4749146Z 
2022-09-27T15:46:25.4749641Z ----------------------------------------------------------------------
2022-09-27T15:46:25.4750074Z Ran 1 test in 2.253s
2022-09-27T15:46:25.4750196Z 
2022-09-27T15:46:25.4750257Z OK
2022-09-27T15:46:25.4750352Z 
2022-09-27T15:46:25.4750433Z Generating XML reports...
2022-09-27T15:46:25.4795435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154623.xml
2022-09-27T15:46:26.7747673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:26.7748067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:26.7756407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:26.7756890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:27.1139978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:27.1149248Z 
2022-09-27T15:46:27.1149384Z Running tests...
2022-09-27T15:46:27.1149783Z ----------------------------------------------------------------------
2022-09-27T15:46:27.3537607Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15257
2022-09-27T15:46:27.3563817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15258
2022-09-27T15:46:27.3591030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15259
2022-09-27T15:46:28.3873054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:28.3873709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:28.4189731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:28.4190404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:28.4246563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:28.4247199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:28.4341909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:28.4342565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:28.4599931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:28.4600656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:28.4685282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:28.4685912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:28.7971697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:28.8048242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:28.8048816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:28.8049449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:28.8049997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:28.8072611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:28.8155556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:28.8156108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:28.9081082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:28.9387654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:46:28.9489927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:46:28.9490975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:28.9491634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:46:28.9492496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:28.9493262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:29.2640876Z ok (2.149s)
2022-09-27T15:46:29.2641091Z 
2022-09-27T15:46:29.2641564Z ----------------------------------------------------------------------
2022-09-27T15:46:29.2641993Z Ran 1 test in 2.149s
2022-09-27T15:46:29.2642206Z 
2022-09-27T15:46:29.2642319Z OK
2022-09-27T15:46:29.2642440Z 
2022-09-27T15:46:29.2642534Z Generating XML reports...
2022-09-27T15:46:29.2689001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154627.xml
2022-09-27T15:46:30.6005148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:30.6005527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:30.6012497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:30.6012983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:30.9578660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:30.9588416Z 
2022-09-27T15:46:30.9588527Z Running tests...
2022-09-27T15:46:30.9589230Z ----------------------------------------------------------------------
2022-09-27T15:46:31.2047840Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15319
2022-09-27T15:46:31.2074241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15320
2022-09-27T15:46:31.2101957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15321
2022-09-27T15:46:32.3199869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:32.3200251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:32.3289680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:32.3290087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:32.3461716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:32.3462305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:32.3541899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:32.3542397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:32.3648347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:32.3648844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:32.3799184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:32.3799825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:32.7154230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:32.7255511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:32.7256150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:32.7256929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:32.7257631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:32.7258549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:32.7264260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:32.7264877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:32.7265526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:32.7266235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:46:32.7371720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:46:32.7372187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:46:32.7372953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:32.7373600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:32.7469888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:46:33.1154243Z ok (2.156s)
2022-09-27T15:46:33.1154484Z 
2022-09-27T15:46:33.1155022Z ----------------------------------------------------------------------
2022-09-27T15:46:33.1155389Z Ran 1 test in 2.157s
2022-09-27T15:46:33.1155509Z 
2022-09-27T15:46:33.1155572Z OK
2022-09-27T15:46:33.1155664Z 
2022-09-27T15:46:33.1155745Z Generating XML reports...
2022-09-27T15:46:33.1200661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154630.xml
2022-09-27T15:46:34.4515603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:34.4516264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:34.4524661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:34.4525319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:34.7922226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:34.7932248Z 
2022-09-27T15:46:34.7932545Z Running tests...
2022-09-27T15:46:34.7933154Z ----------------------------------------------------------------------
2022-09-27T15:46:34.7958335Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:46:34.7958689Z 
2022-09-27T15:46:34.7958927Z ----------------------------------------------------------------------
2022-09-27T15:46:34.7959163Z Ran 1 test in 0.003s
2022-09-27T15:46:34.7959279Z 
2022-09-27T15:46:34.7959353Z OK (skipped=1)
2022-09-27T15:46:34.7959463Z 
2022-09-27T15:46:34.7959559Z Generating XML reports...
2022-09-27T15:46:34.7989212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154634.xml
2022-09-27T15:46:35.9802831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:35.9803205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:35.9810927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:35.9811288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:36.3174082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:36.3183261Z 
2022-09-27T15:46:36.3183418Z Running tests...
2022-09-27T15:46:36.3183994Z ----------------------------------------------------------------------
2022-09-27T15:46:36.5522212Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15388
2022-09-27T15:46:36.5548878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15389
2022-09-27T15:46:36.5577548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15390
2022-09-27T15:46:37.6422505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:37.6422909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:37.6733451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:37.6733926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:37.6742091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:37.6742611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:37.6865410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:37.6865847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:37.7076576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:37.7077007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:37.7210403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:37.7210759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:38.0748949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:38.0849484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:38.0850206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:38.0851282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:38.0851875Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:38.0852402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:38.0858058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:38.0859141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:38.0859910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:38.4629552Z ok (2.144s)
2022-09-27T15:46:38.4629780Z 
2022-09-27T15:46:38.4630136Z ----------------------------------------------------------------------
2022-09-27T15:46:38.4630388Z Ran 1 test in 2.145s
2022-09-27T15:46:38.4630505Z 
2022-09-27T15:46:38.4630552Z OK
2022-09-27T15:46:38.4630678Z 
2022-09-27T15:46:38.4630781Z Generating XML reports...
2022-09-27T15:46:38.4675795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154636.xml
2022-09-27T15:46:39.7917722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:39.7918106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:39.7925911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:39.7926277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:40.1298889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:40.1308734Z 
2022-09-27T15:46:40.1308821Z Running tests...
2022-09-27T15:46:40.1309808Z ----------------------------------------------------------------------
2022-09-27T15:46:40.3643379Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15441
2022-09-27T15:46:40.3669400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15442
2022-09-27T15:46:40.3696189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15443
2022-09-27T15:46:41.4603819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:41.4604536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:41.4838813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:41.4839502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:41.4857119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:41.4857758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:41.4937226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:41.4937862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:41.5195405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:41.5196094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:41.5214954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:41.5215625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:41.8623304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:41.8651064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:41.8651659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:41.8652734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:41.8653298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:41.8724280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:41.8758210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:41.8758959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:41.9733045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:42.3751230Z ok (2.244s)
2022-09-27T15:46:42.3751803Z 
2022-09-27T15:46:42.3752711Z ----------------------------------------------------------------------
2022-09-27T15:46:42.3753131Z Ran 1 test in 2.244s
2022-09-27T15:46:42.3753320Z 
2022-09-27T15:46:42.3753418Z OK
2022-09-27T15:46:42.3753576Z 
2022-09-27T15:46:42.3753739Z Generating XML reports...
2022-09-27T15:46:42.3797960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154640.xml
2022-09-27T15:46:43.6976964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:43.6977342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:43.6984697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:43.6985149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:44.0358279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:44.0367592Z 
2022-09-27T15:46:44.0367732Z Running tests...
2022-09-27T15:46:44.0368172Z ----------------------------------------------------------------------
2022-09-27T15:46:44.2715915Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15494
2022-09-27T15:46:44.2741388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15495
2022-09-27T15:46:44.2768863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15496
2022-09-27T15:46:45.3217761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:45.3218356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:45.3379392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:45.3379878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:45.3520444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:45.3520957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:45.3564044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:45.3564571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:45.3733199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:45.3733845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:45.3859967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:45.3860593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:45.7271976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:45.7372919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:45.7373395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:45.7374007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:45.7374559Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:45.7375071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:45.7381119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:45.7384368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:45.7385204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:45.7386014Z STAGE:2022-09-27 15:46:45 15494:15494 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:45.7386764Z STAGE:2022-09-27 15:46:45 15496:15496 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:45.7388486Z STAGE:2022-09-27 15:46:45 15495:15495 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:45.7437929Z STAGE:2022-09-27 15:46:45 15495:15495 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:45.8421164Z STAGE:2022-09-27 15:46:45 15494:15494 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:45.8423629Z STAGE:2022-09-27 15:46:45 15496:15496 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:46.1815972Z ok (2.144s)
2022-09-27T15:46:46.1816268Z 
2022-09-27T15:46:46.1816712Z ----------------------------------------------------------------------
2022-09-27T15:46:46.1817114Z Ran 1 test in 2.145s
2022-09-27T15:46:46.1817302Z 
2022-09-27T15:46:46.1817400Z OK
2022-09-27T15:46:46.1817541Z 
2022-09-27T15:46:46.1817663Z Generating XML reports...
2022-09-27T15:46:46.1863751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154644.xml
2022-09-27T15:46:47.5296948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:47.5297316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:47.5305347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:47.5305906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:47.8831293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:47.8840939Z 
2022-09-27T15:46:47.8841069Z Running tests...
2022-09-27T15:46:47.8842009Z ----------------------------------------------------------------------
2022-09-27T15:46:48.1246102Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15550
2022-09-27T15:46:48.1272472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15551
2022-09-27T15:46:48.1300134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15552
2022-09-27T15:46:49.2296699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:49.2297421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:49.2398427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:49.2399051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:49.2643753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:49.2644174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:49.2644681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:49.2645093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:49.2756423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:49.2756797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:49.2965305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:49.2965674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:49.6221257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:49.6321496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:49.6322218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:49.6323418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:49.6324037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:49.6324724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:49.6330994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:49.6335944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:49.6336380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:49.6336912Z STAGE:2022-09-27 15:46:49 15550:15550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:49.6337544Z STAGE:2022-09-27 15:46:49 15551:15551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:49.6338384Z STAGE:2022-09-27 15:46:49 15552:15552 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:46:49.6375677Z STAGE:2022-09-27 15:46:49 15551:15551 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:49.6376701Z STAGE:2022-09-27 15:46:49 15552:15552 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:49.6377493Z STAGE:2022-09-27 15:46:49 15550:15550 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:46:50.0349070Z ok (2.150s)
2022-09-27T15:46:50.0349233Z 
2022-09-27T15:46:50.0350078Z ----------------------------------------------------------------------
2022-09-27T15:46:50.0350554Z Ran 1 test in 2.151s
2022-09-27T15:46:50.0350741Z 
2022-09-27T15:46:50.0350842Z OK
2022-09-27T15:46:50.0350985Z 
2022-09-27T15:46:50.0351139Z Generating XML reports...
2022-09-27T15:46:50.0396587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154647.xml
2022-09-27T15:46:51.3457695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:51.3458081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:51.3466813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:51.3467173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:51.6851533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:51.6861058Z 
2022-09-27T15:46:51.6861154Z Running tests...
2022-09-27T15:46:51.6862108Z ----------------------------------------------------------------------
2022-09-27T15:46:51.6877679Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:46:51.6878075Z 
2022-09-27T15:46:51.6878420Z ----------------------------------------------------------------------
2022-09-27T15:46:51.6878885Z Ran 1 test in 0.002s
2022-09-27T15:46:51.6879098Z 
2022-09-27T15:46:51.6879191Z OK (skipped=1)
2022-09-27T15:46:51.6879309Z 
2022-09-27T15:46:51.6879403Z Generating XML reports...
2022-09-27T15:46:51.6908644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154651.xml
2022-09-27T15:46:52.8789887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:52.8790272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:52.8797933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:52.8798365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:53.2223440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:53.2233255Z 
2022-09-27T15:46:53.2233764Z Running tests...
2022-09-27T15:46:53.2234312Z ----------------------------------------------------------------------
2022-09-27T15:46:53.2250496Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:46:53.2250796Z 
2022-09-27T15:46:53.2251154Z ----------------------------------------------------------------------
2022-09-27T15:46:53.2251595Z Ran 1 test in 0.002s
2022-09-27T15:46:53.2251780Z 
2022-09-27T15:46:53.2251863Z OK (skipped=1)
2022-09-27T15:46:53.2251971Z 
2022-09-27T15:46:53.2252057Z Generating XML reports...
2022-09-27T15:46:53.2282553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154653.xml
2022-09-27T15:46:54.4163605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:54.4164229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:54.4171311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:54.4171892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:54.7549043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:54.7559264Z 
2022-09-27T15:46:54.7559530Z Running tests...
2022-09-27T15:46:54.7560181Z ----------------------------------------------------------------------
2022-09-27T15:46:54.9921582Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15626
2022-09-27T15:46:54.9948099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15627
2022-09-27T15:46:54.9975334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15628
2022-09-27T15:46:56.0644554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:56.0645237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:56.0961677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:56.0962250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:56.0963813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:56.0964373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:56.1072620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:56.1073233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:56.1307190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:56.1307881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:56.1410318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:56.1410980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:56.4770915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:56.4771579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:56.4772207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:46:56.4772870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:56.4773399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:56.4773925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:46:56.4875983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:56.5779817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:56.5781193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:46:58.5800302Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T15:46:58.5898891Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-09-27T15:46:59.0052964Z ok (4.249s)
2022-09-27T15:46:59.0053222Z 
2022-09-27T15:46:59.0053928Z ----------------------------------------------------------------------
2022-09-27T15:46:59.0054333Z Ran 1 test in 4.249s
2022-09-27T15:46:59.0054496Z 
2022-09-27T15:46:59.0054587Z OK
2022-09-27T15:46:59.0054835Z 
2022-09-27T15:46:59.0054990Z Generating XML reports...
2022-09-27T15:46:59.0100348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154654.xml
2022-09-27T15:47:00.3087845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:00.3088281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:00.3096000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:00.3096370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:00.6471339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:00.6481034Z 
2022-09-27T15:47:00.6481120Z Running tests...
2022-09-27T15:47:00.6481917Z ----------------------------------------------------------------------
2022-09-27T15:47:00.8833573Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15679
2022-09-27T15:47:00.8860055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15680
2022-09-27T15:47:00.8886887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15681
2022-09-27T15:47:01.9713798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:01.9714422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:01.9748232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:01.9748895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:01.9831776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:01.9832307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:02.0064021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:02.0064430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:02.0104324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:02.0104808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:02.0184108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:02.0184626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:02.3469126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:02.3469825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:02.3470199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:02.3470816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:02.3471350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:02.3471878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:02.3477865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:02.3478799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:02.3479621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:04.3679752Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T15:47:04.3680385Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T15:47:04.7966105Z ok (4.148s)
2022-09-27T15:47:04.7966407Z 
2022-09-27T15:47:04.7966926Z ----------------------------------------------------------------------
2022-09-27T15:47:04.7967171Z Ran 1 test in 4.148s
2022-09-27T15:47:04.7967291Z 
2022-09-27T15:47:04.7967354Z OK
2022-09-27T15:47:04.7967449Z 
2022-09-27T15:47:04.7967544Z Generating XML reports...
2022-09-27T15:47:04.8012329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154700.xml
2022-09-27T15:47:06.1236018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:06.1236422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:06.1244110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:06.1244480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:06.4624236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:06.4633867Z 
2022-09-27T15:47:06.4633994Z Running tests...
2022-09-27T15:47:06.4634461Z ----------------------------------------------------------------------
2022-09-27T15:47:06.7015003Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15732
2022-09-27T15:47:06.7042171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15733
2022-09-27T15:47:06.7069514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15734
2022-09-27T15:47:07.7918028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:07.7918659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:07.8038245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:07.8038924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:07.8105985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:07.8106506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:07.8267200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:07.8267768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:07.8393486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:07.8394147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:07.8452882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:07.8453500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:08.1709832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:08.1810797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:08.1811330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:08.1812357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:08.1812902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:08.1813497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:08.1819070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:08.1819645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:08.1820152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:08.2024359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:08.2126166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:47:08.2126874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:08.2127624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:08.2128579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:08.2227629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:08.2231519Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-09-27T15:47:08.2232072Z No ranks successfully processed in monitoredBarrier.
2022-09-27T15:47:08.2257831Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-09-27T15:47:08.6119192Z ok (2.148s)
2022-09-27T15:47:08.6119372Z 
2022-09-27T15:47:08.6119894Z ----------------------------------------------------------------------
2022-09-27T15:47:08.6120362Z Ran 1 test in 2.148s
2022-09-27T15:47:08.6120570Z 
2022-09-27T15:47:08.6120686Z OK
2022-09-27T15:47:08.6120866Z 
2022-09-27T15:47:08.6121010Z Generating XML reports...
2022-09-27T15:47:08.6164875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml
2022-09-27T15:47:09.9399463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:09.9399895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:09.9406879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:09.9407415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:10.2770195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:10.2779467Z 
2022-09-27T15:47:10.2779613Z Running tests...
2022-09-27T15:47:10.2780244Z ----------------------------------------------------------------------
2022-09-27T15:47:10.5135633Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15794
2022-09-27T15:47:10.5162693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15795
2022-09-27T15:47:10.5190323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15796
2022-09-27T15:47:11.6146160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:11.6146555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:11.6242018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:11.6344048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:11.6344644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:11.6345136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:11.6490686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:11.6491052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:11.6601467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:11.6601839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:11.6695762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:11.6696393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.0185367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:12.0285727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:12.0286334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:12.0287292Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:12.0288202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:12.0288807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:12.0393058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:12.0394119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:12.0394791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:47:12.1295643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:12.1404878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:12.1405339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:12.1406018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:12.1406564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:12.1506221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:47:12.1511553Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:350: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-09-27T15:47:12.1511994Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-09-27T15:47:12.2510849Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T15:47:12.6243106Z ok (2.346s)
2022-09-27T15:47:12.6243346Z 
2022-09-27T15:47:12.6243857Z ----------------------------------------------------------------------
2022-09-27T15:47:12.6244245Z Ran 1 test in 2.346s
2022-09-27T15:47:12.6244364Z 
2022-09-27T15:47:12.6244429Z OK
2022-09-27T15:47:12.6244508Z 
2022-09-27T15:47:12.6244602Z Generating XML reports...
2022-09-27T15:47:12.6290966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154710.xml
2022-09-27T15:47:13.9700693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:13.9701085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:13.9709396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:13.9709965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:14.3284681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:14.3294274Z 
2022-09-27T15:47:14.3294416Z Running tests...
2022-09-27T15:47:14.3295006Z ----------------------------------------------------------------------
2022-09-27T15:47:14.5690032Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15853
2022-09-27T15:47:14.5717226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15854
2022-09-27T15:47:14.5745998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15855
2022-09-27T15:47:15.6462123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.6462766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.6605971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.6606357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.6796401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.6796799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:15.6943069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.6943451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:15.7059755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.7060296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.7379349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.7379864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:16.0817129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:16.0918430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:16.0918889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:16.0919513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:16.0920639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:16.0921153Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:16.0926618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:16.0927129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:16.0928122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:16.2123366Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T15:47:16.2124037Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-09-27T15:47:16.2124585Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-09-27T15:47:16.5796719Z ok (2.250s)
2022-09-27T15:47:16.5796931Z 
2022-09-27T15:47:16.5797459Z ----------------------------------------------------------------------
2022-09-27T15:47:16.5797964Z Ran 1 test in 2.250s
2022-09-27T15:47:16.5798081Z 
2022-09-27T15:47:16.5798144Z OK
2022-09-27T15:47:16.5798242Z 
2022-09-27T15:47:16.5798335Z Generating XML reports...
2022-09-27T15:47:16.5844372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154714.xml
2022-09-27T15:47:17.9125821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:17.9126451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:17.9134301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:17.9134952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:18.2521563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:18.2530915Z 
2022-09-27T15:47:18.2531143Z Running tests...
2022-09-27T15:47:18.2531496Z ----------------------------------------------------------------------
2022-09-27T15:47:18.2553398Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T15:47:18.2553752Z 
2022-09-27T15:47:18.2553969Z ----------------------------------------------------------------------
2022-09-27T15:47:18.2554256Z Ran 1 test in 0.002s
2022-09-27T15:47:18.2554421Z 
2022-09-27T15:47:18.2554504Z OK (skipped=1)
2022-09-27T15:47:18.2554612Z 
2022-09-27T15:47:18.2554686Z Generating XML reports...
2022-09-27T15:47:18.2584909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154718.xml
2022-09-27T15:47:19.4448040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.4448427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.4455593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.4456127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.7819784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:19.7829532Z 
2022-09-27T15:47:19.7829681Z Running tests...
2022-09-27T15:47:19.7830336Z ----------------------------------------------------------------------
2022-09-27T15:47:19.7852067Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T15:47:19.7852340Z 
2022-09-27T15:47:19.7852605Z ----------------------------------------------------------------------
2022-09-27T15:47:19.7852870Z Ran 1 test in 0.002s
2022-09-27T15:47:19.7852988Z 
2022-09-27T15:47:19.7853065Z OK (skipped=1)
2022-09-27T15:47:19.7853185Z 
2022-09-27T15:47:19.7853327Z Generating XML reports...
2022-09-27T15:47:19.7884052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154719.xml
2022-09-27T15:47:20.9775313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:20.9775930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:20.9783820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:20.9784517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:21.3201856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:21.3210989Z 
2022-09-27T15:47:21.3211843Z Running tests...
2022-09-27T15:47:21.3212338Z ----------------------------------------------------------------------
2022-09-27T15:47:21.3232581Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T15:47:21.3233219Z 
2022-09-27T15:47:21.3233620Z ----------------------------------------------------------------------
2022-09-27T15:47:21.3233911Z Ran 1 test in 0.002s
2022-09-27T15:47:21.3234029Z 
2022-09-27T15:47:21.3234105Z OK (skipped=1)
2022-09-27T15:47:21.3234216Z 
2022-09-27T15:47:21.3234288Z Generating XML reports...
2022-09-27T15:47:21.3264075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154721.xml
2022-09-27T15:47:22.5172196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:22.5172837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:22.5181258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:22.5181883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:22.8555837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:22.8566141Z 
2022-09-27T15:47:22.8566597Z Running tests...
2022-09-27T15:47:22.8567015Z ----------------------------------------------------------------------
2022-09-27T15:47:22.8593214Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T15:47:22.8593536Z 
2022-09-27T15:47:22.8593774Z ----------------------------------------------------------------------
2022-09-27T15:47:22.8594028Z Ran 1 test in 0.003s
2022-09-27T15:47:22.8594143Z 
2022-09-27T15:47:22.8594217Z OK (skipped=1)
2022-09-27T15:47:22.8594328Z 
2022-09-27T15:47:22.8594401Z Generating XML reports...
2022-09-27T15:47:22.8622298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154722.xml
2022-09-27T15:47:24.0633700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:24.0634351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:24.0643175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:24.0643780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:24.4039197Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:24.4048225Z 
2022-09-27T15:47:24.4048339Z Running tests...
2022-09-27T15:47:24.4048914Z ----------------------------------------------------------------------
2022-09-27T15:47:24.4070674Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T15:47:24.4071093Z 
2022-09-27T15:47:24.4071416Z ----------------------------------------------------------------------
2022-09-27T15:47:24.4071660Z Ran 1 test in 0.002s
2022-09-27T15:47:24.4071772Z 
2022-09-27T15:47:24.4071847Z OK (skipped=1)
2022-09-27T15:47:24.4071959Z 
2022-09-27T15:47:24.4072044Z Generating XML reports...
2022-09-27T15:47:24.4100794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154724.xml
2022-09-27T15:47:25.5895794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:25.5896188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:25.5904031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:25.5904626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:25.9272302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:25.9281380Z 
2022-09-27T15:47:25.9281650Z Running tests...
2022-09-27T15:47:25.9282291Z ----------------------------------------------------------------------
2022-09-27T15:47:25.9300504Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T15:47:25.9300814Z 
2022-09-27T15:47:25.9301078Z ----------------------------------------------------------------------
2022-09-27T15:47:25.9301332Z Ran 1 test in 0.002s
2022-09-27T15:47:25.9301435Z 
2022-09-27T15:47:25.9301537Z OK (skipped=1)
2022-09-27T15:47:25.9301677Z 
2022-09-27T15:47:25.9301764Z Generating XML reports...
2022-09-27T15:47:25.9331375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154725.xml
2022-09-27T15:47:27.1180332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:27.1180704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:27.1188788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:27.1189280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:27.4582361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:27.4591976Z 
2022-09-27T15:47:27.4592118Z Running tests...
2022-09-27T15:47:27.4592717Z ----------------------------------------------------------------------
2022-09-27T15:47:27.4613982Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T15:47:27.4614342Z 
2022-09-27T15:47:27.4614703Z ----------------------------------------------------------------------
2022-09-27T15:47:27.4615154Z Ran 1 test in 0.002s
2022-09-27T15:47:27.4615353Z 
2022-09-27T15:47:27.4615485Z OK (skipped=1)
2022-09-27T15:47:27.4615659Z 
2022-09-27T15:47:27.4615811Z Generating XML reports...
2022-09-27T15:47:27.4645257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154727.xml
2022-09-27T15:47:28.6531620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:28.6531994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:28.6539176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:28.6539543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:28.9906432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:28.9916376Z 
2022-09-27T15:47:28.9916702Z Running tests...
2022-09-27T15:47:28.9917121Z ----------------------------------------------------------------------
2022-09-27T15:47:28.9935341Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T15:47:28.9935791Z 
2022-09-27T15:47:28.9936219Z ----------------------------------------------------------------------
2022-09-27T15:47:28.9936633Z Ran 1 test in 0.002s
2022-09-27T15:47:28.9936752Z 
2022-09-27T15:47:28.9936826Z OK (skipped=1)
2022-09-27T15:47:28.9936933Z 
2022-09-27T15:47:28.9937022Z Generating XML reports...
2022-09-27T15:47:28.9966469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154728.xml
2022-09-27T15:47:30.1808716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:30.1809110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:30.1817021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:30.1817411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:30.5215727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:30.5225075Z 
2022-09-27T15:47:30.5225231Z Running tests...
2022-09-27T15:47:30.5225857Z ----------------------------------------------------------------------
2022-09-27T15:47:30.7588795Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15986
2022-09-27T15:47:30.7616241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15987
2022-09-27T15:47:30.7643242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15988
2022-09-27T15:47:31.8453929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:31.8454525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:31.8685414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:31.8686032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:31.8779754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:31.8780161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:31.9015925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:31.9016463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:31.9433353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:31.9433822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:31.9750376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:31.9750813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:32.3316381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:32.3416810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:32.3417366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:32.3418088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:32.3418650Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:32.3419168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:32.3524574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:32.3524969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:32.4430359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:32.6694004Z skip: CUDA is not available. (2.146s)
2022-09-27T15:47:32.6694286Z 
2022-09-27T15:47:32.6694709Z ----------------------------------------------------------------------
2022-09-27T15:47:32.6695106Z Ran 1 test in 2.147s
2022-09-27T15:47:32.6695280Z 
2022-09-27T15:47:32.6695394Z OK (skipped=1)
2022-09-27T15:47:32.6695568Z 
2022-09-27T15:47:32.6695715Z Generating XML reports...
2022-09-27T15:47:32.6741856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154730.xml
2022-09-27T15:47:33.9555910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:33.9556517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:33.9563576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:33.9564108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:34.2950082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:34.2958619Z 
2022-09-27T15:47:34.2958762Z Running tests...
2022-09-27T15:47:34.2959352Z ----------------------------------------------------------------------
2022-09-27T15:47:34.5328093Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16039
2022-09-27T15:47:34.5355481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16040
2022-09-27T15:47:34.5382423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16041
2022-09-27T15:47:35.6412874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:35.6413524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:35.6707542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:35.6708236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:35.6730170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:35.6730825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:35.7034114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:35.7034570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:35.7036559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:35.7036905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:35.7351979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:35.7352348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:36.0569913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:36.0570633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:36.0571093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:36.0571705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:36.0572233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:36.0572757Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:36.0577869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:36.0578751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:36.0580990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:36.3431874Z skip: CUDA is not available. (2.047s)
2022-09-27T15:47:36.3432213Z 
2022-09-27T15:47:36.3432913Z ----------------------------------------------------------------------
2022-09-27T15:47:36.3433162Z Ran 1 test in 2.047s
2022-09-27T15:47:36.3433282Z 
2022-09-27T15:47:36.3433359Z OK (skipped=1)
2022-09-27T15:47:36.3433542Z 
2022-09-27T15:47:36.3433630Z Generating XML reports...
2022-09-27T15:47:36.3478298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154734.xml
2022-09-27T15:47:37.6293954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:37.6294638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:37.6302924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:37.6303604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:37.9661498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:37.9671446Z 
2022-09-27T15:47:37.9671929Z Running tests...
2022-09-27T15:47:37.9672329Z ----------------------------------------------------------------------
2022-09-27T15:47:37.9689656Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T15:47:37.9690109Z 
2022-09-27T15:47:37.9690403Z ----------------------------------------------------------------------
2022-09-27T15:47:37.9690642Z Ran 1 test in 0.002s
2022-09-27T15:47:37.9690761Z 
2022-09-27T15:47:37.9690834Z OK (skipped=1)
2022-09-27T15:47:37.9690944Z 
2022-09-27T15:47:37.9691032Z Generating XML reports...
2022-09-27T15:47:37.9720724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154737.xml
2022-09-27T15:47:39.1596066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:39.1596506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:39.1604111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:39.1604509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:39.4969196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:39.4978386Z 
2022-09-27T15:47:39.4978482Z Running tests...
2022-09-27T15:47:39.4979149Z ----------------------------------------------------------------------
2022-09-27T15:47:39.4996221Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s)
2022-09-27T15:47:39.4996531Z 
2022-09-27T15:47:39.4996760Z ----------------------------------------------------------------------
2022-09-27T15:47:39.4997052Z Ran 1 test in 0.002s
2022-09-27T15:47:39.4997169Z 
2022-09-27T15:47:39.4997245Z OK (skipped=1)
2022-09-27T15:47:39.4997370Z 
2022-09-27T15:47:39.4997457Z Generating XML reports...
2022-09-27T15:47:39.5026466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154739.xml
2022-09-27T15:47:40.6975675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:40.6976362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:40.6984859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:40.6985582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:41.0364054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:41.0373917Z 
2022-09-27T15:47:41.0374368Z Running tests...
2022-09-27T15:47:41.0374783Z ----------------------------------------------------------------------
2022-09-27T15:47:41.2674428Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.230s)
2022-09-27T15:47:41.2675064Z 
2022-09-27T15:47:41.2675278Z ----------------------------------------------------------------------
2022-09-27T15:47:41.2675609Z Ran 1 test in 0.230s
2022-09-27T15:47:41.2675725Z 
2022-09-27T15:47:41.2675785Z OK (skipped=1)
2022-09-27T15:47:41.2675894Z 
2022-09-27T15:47:41.2675979Z Generating XML reports...
2022-09-27T15:47:41.2697241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154741.xml
2022-09-27T15:47:42.5092547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:42.5092929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:42.5100872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:42.5101331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:42.8505241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:42.8514501Z 
2022-09-27T15:47:42.8514619Z Running tests...
2022-09-27T15:47:42.8515238Z ----------------------------------------------------------------------
2022-09-27T15:47:43.0883960Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16122
2022-09-27T15:47:43.0911513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16123
2022-09-27T15:47:43.0938778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16124
2022-09-27T15:47:44.1950720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:44.1951400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:44.2107470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:44.2108078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:44.2183247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:44.2184097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:44.2290080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:44.2290614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:44.2460708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:44.2461121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:44.2533489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:44.2533958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:44.5760211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:44.5760859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:44.5761248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:44.5762030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:44.5762580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:44.5763167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:44.5865785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:44.6774442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:44.9991385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:44.9991959Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:47:44.9992241Z 
2022-09-27T15:47:44.9992624Z ----------------------------------------------------------------------
2022-09-27T15:47:44.9992862Z Ran 1 test in 2.148s
2022-09-27T15:47:44.9992978Z 
2022-09-27T15:47:44.9993053Z OK (skipped=1)
2022-09-27T15:47:44.9993181Z 
2022-09-27T15:47:44.9993270Z Generating XML reports...
2022-09-27T15:47:45.0039429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154742.xml
2022-09-27T15:47:46.3444615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:46.3444999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:46.3452771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:46.3453275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:46.7018580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:46.7028243Z 
2022-09-27T15:47:46.7028382Z Running tests...
2022-09-27T15:47:46.7028963Z ----------------------------------------------------------------------
2022-09-27T15:47:46.9573030Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16175
2022-09-27T15:47:46.9599218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16176
2022-09-27T15:47:46.9627499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16177
2022-09-27T15:47:48.0369039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:48.0369630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:48.0693242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:48.0693876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:48.0703754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:48.0704307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:48.0727886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:48.0728490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:48.1056213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:48.1056611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:48.1080431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:48.1080897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:48.4353121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:48.4454705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:48.4455456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:48.4456591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:48.4457155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:48.4457687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:48.4461909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:48.4465343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:48.4465717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:48.7677352Z skip: Need at least 2 CUDA devices (2.064s)
2022-09-27T15:47:48.7677755Z 
2022-09-27T15:47:48.7678184Z ----------------------------------------------------------------------
2022-09-27T15:47:48.7678438Z Ran 1 test in 2.065s
2022-09-27T15:47:48.7678556Z 
2022-09-27T15:47:48.7678631Z OK (skipped=1)
2022-09-27T15:47:48.7678741Z 
2022-09-27T15:47:48.7678830Z Generating XML reports...
2022-09-27T15:47:48.7724292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154746.xml
2022-09-27T15:47:50.0564559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:50.0564955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:50.0572669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:50.0573205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:50.3947231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:50.3956699Z 
2022-09-27T15:47:50.3956857Z Running tests...
2022-09-27T15:47:50.3957940Z ----------------------------------------------------------------------
2022-09-27T15:47:50.6314690Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16228
2022-09-27T15:47:50.6340921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16229
2022-09-27T15:47:50.6367907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16230
2022-09-27T15:47:51.6905761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:51.6906177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:51.7216487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:51.7216859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:51.7491373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:51.7491738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:51.7672745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:51.7673289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:51.7827006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:51.7827433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:51.8010308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:51.8010990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:52.1184748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:52.1272497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:52.1272965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:52.1273644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:52.1274171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:52.1285800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:52.1379199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:52.1379795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:52.2297767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:52.5417255Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T15:47:52.5417550Z 
2022-09-27T15:47:52.5418071Z ----------------------------------------------------------------------
2022-09-27T15:47:52.5418381Z Ran 1 test in 2.146s
2022-09-27T15:47:52.5418484Z 
2022-09-27T15:47:52.5418559Z OK (skipped=1)
2022-09-27T15:47:52.5418667Z 
2022-09-27T15:47:52.5418757Z Generating XML reports...
2022-09-27T15:47:52.5463771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154750.xml
2022-09-27T15:47:53.8601115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:53.8601779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:53.8610095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:53.8610773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:54.1993766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:54.2003894Z 
2022-09-27T15:47:54.2004385Z Running tests...
2022-09-27T15:47:54.2004827Z ----------------------------------------------------------------------
2022-09-27T15:47:54.4352289Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16281
2022-09-27T15:47:54.4379000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16282
2022-09-27T15:47:54.4406291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16283
2022-09-27T15:47:55.4792049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:55.4792682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:55.4980680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:55.4981341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:55.5124745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:55.5125289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:55.5314116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:55.5314519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:55.5667582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:55.5668015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:55.5980261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:55.5980643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:55.9198934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:55.9296563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:55.9297054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:55.9297677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:55.9298207Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:55.9299898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:55.9403098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:55.9403930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:56.0311693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:56.3456389Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:47:56.3456582Z 
2022-09-27T15:47:56.3456894Z ----------------------------------------------------------------------
2022-09-27T15:47:56.3457147Z Ran 1 test in 2.145s
2022-09-27T15:47:56.3457262Z 
2022-09-27T15:47:56.3457322Z OK (skipped=1)
2022-09-27T15:47:56.3457432Z 
2022-09-27T15:47:56.3457535Z Generating XML reports...
2022-09-27T15:47:56.3502259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154754.xml
2022-09-27T15:47:57.6497624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:57.6498005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:57.6505737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:57.6506101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:57.9870860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:57.9880414Z 
2022-09-27T15:47:57.9880536Z Running tests...
2022-09-27T15:47:57.9881091Z ----------------------------------------------------------------------
2022-09-27T15:47:58.2244475Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16334
2022-09-27T15:47:58.2271383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16335
2022-09-27T15:47:58.2298162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16336
2022-09-27T15:47:59.2903234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:59.2903605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:59.2959283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:59.2959930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:59.3247714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:59.3248646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:59.3308927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:59.3309802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:59.3456294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:59.3456872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:59.3768963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:59.3769368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:59.6916478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:59.7010251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:59.7010661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:47:59.7011305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:59.7011934Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:59.7017178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:47:59.7117329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:59.7118479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:47:59.8028912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:00.0348478Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:48:00.0348732Z 
2022-09-27T15:48:00.0349070Z ----------------------------------------------------------------------
2022-09-27T15:48:00.0349337Z Ran 1 test in 2.047s
2022-09-27T15:48:00.0349472Z 
2022-09-27T15:48:00.0349567Z OK (skipped=1)
2022-09-27T15:48:00.0349678Z 
2022-09-27T15:48:00.0349765Z Generating XML reports...
2022-09-27T15:48:00.0395063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154757.xml
2022-09-27T15:48:01.3288334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:01.3288778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:01.3296641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:01.3296988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:01.6696996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:01.6706852Z 
2022-09-27T15:48:01.6706965Z Running tests...
2022-09-27T15:48:01.6707390Z ----------------------------------------------------------------------
2022-09-27T15:48:01.9085112Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16387
2022-09-27T15:48:01.9111367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16388
2022-09-27T15:48:01.9137859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16389
2022-09-27T15:48:02.9865396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:02.9865765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:02.9941538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:02.9941890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:03.0216934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:03.0217293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:03.0295346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:03.0295754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:03.0311832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:03.0312167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:03.0648909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:03.0649293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:03.4059290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:03.4159375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:03.4159926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:03.4160743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:03.4161283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:03.4161805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:03.4267137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:03.4267951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:03.5173026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:03.8186264Z skip: Need at least 4 CUDA devices (2.148s)
2022-09-27T15:48:03.8186548Z 
2022-09-27T15:48:03.8187040Z ----------------------------------------------------------------------
2022-09-27T15:48:03.8187462Z Ran 1 test in 2.148s
2022-09-27T15:48:03.8187578Z 
2022-09-27T15:48:03.8187652Z OK (skipped=1)
2022-09-27T15:48:03.8187759Z 
2022-09-27T15:48:03.8187850Z Generating XML reports...
2022-09-27T15:48:03.8233028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154801.xml
2022-09-27T15:48:05.1240888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:05.1241267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:05.1248598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:05.1249024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:05.4635761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:05.4645522Z 
2022-09-27T15:48:05.4645979Z Running tests...
2022-09-27T15:48:05.4646598Z ----------------------------------------------------------------------
2022-09-27T15:48:05.6997279Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16440
2022-09-27T15:48:05.7023491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16441
2022-09-27T15:48:05.7051024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16442
2022-09-27T15:48:06.7661323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:06.7661927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:06.7988549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:06.7988975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:06.7992429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:06.7992987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:06.8313840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:06.8314246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:06.8422908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:06.8423242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:06.8742041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:06.8742398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:07.2089109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:07.2190503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:07.2191114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:07.2191915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:07.2192471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:07.2193283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:07.2198536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:07.2199362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:07.2200911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:07.5098520Z skip: Need at least 4 CUDA devices (2.045s)
2022-09-27T15:48:07.5098902Z 
2022-09-27T15:48:07.5099293Z ----------------------------------------------------------------------
2022-09-27T15:48:07.5099561Z Ran 1 test in 2.045s
2022-09-27T15:48:07.5099677Z 
2022-09-27T15:48:07.5099750Z OK (skipped=1)
2022-09-27T15:48:07.5099861Z 
2022-09-27T15:48:07.5099953Z Generating XML reports...
2022-09-27T15:48:07.5145423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154805.xml
2022-09-27T15:48:08.7954143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:08.7954512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:08.7962159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:08.7962551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:09.1344963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:09.1354955Z 
2022-09-27T15:48:09.1355060Z Running tests...
2022-09-27T15:48:09.1355527Z ----------------------------------------------------------------------
2022-09-27T15:48:09.3710472Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16493
2022-09-27T15:48:09.3736067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16494
2022-09-27T15:48:09.3763332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16495
2022-09-27T15:48:10.4585218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:10.4585614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:10.4670437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:10.4670918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:10.4845830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:10.4846368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:10.4930029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:10.4930586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:10.5026229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:10.5026833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:10.5198406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:10.5198767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:10.8648684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:10.8748737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:10.8749337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:10.8750030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:10.8750664Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:10.8751188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:10.8857113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:10.8857682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:10.9762368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:11.2810546Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:48:11.2810719Z 
2022-09-27T15:48:11.2811065Z ----------------------------------------------------------------------
2022-09-27T15:48:11.2811328Z Ran 1 test in 2.145s
2022-09-27T15:48:11.2811442Z 
2022-09-27T15:48:11.2811515Z OK (skipped=1)
2022-09-27T15:48:11.2811669Z 
2022-09-27T15:48:11.2811765Z Generating XML reports...
2022-09-27T15:48:11.2856737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154809.xml
2022-09-27T15:48:12.5809117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:12.5809501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:12.5817758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:12.5818109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:12.9240493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:12.9249427Z 
2022-09-27T15:48:12.9249571Z Running tests...
2022-09-27T15:48:12.9250149Z ----------------------------------------------------------------------
2022-09-27T15:48:13.1627685Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16546
2022-09-27T15:48:13.1655039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16547
2022-09-27T15:48:13.1683233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16548
2022-09-27T15:48:14.2724856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:14.2725460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:14.2732913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:14.2733543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:14.3073044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:14.3073460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:14.3084778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:14.3085444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:14.3371799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:14.3372398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:14.3689589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:14.3690250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:14.7221228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:14.7323377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:14.7324063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:14.7324566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:14.7325120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:14.7325763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:14.7331259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:14.7332311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:14.8332468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:14.8443810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:14.8545537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:14.8546466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:14.8547380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:14.8548296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:14.8549259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:14.8550206Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8551062Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8551506Z STAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8557699Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8558580Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8558873Z 
2022-09-27T15:48:14.8565031Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8567942Z STAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8573746Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8577832Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8578559Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8579005Z STAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8584277Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8586612Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8593021Z STAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:14.8598150Z STAGE:2022-09-27 15:48:14 16548:16548 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8598788Z STAGE:2022-09-27 15:48:14 16546:16546 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:14 16547:16547 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:14.8599093Z 
2022-09-27T15:48:15.2734008Z ok (2.348s)
2022-09-27T15:48:15.2734251Z 
2022-09-27T15:48:15.2734764Z ----------------------------------------------------------------------
2022-09-27T15:48:15.2735228Z Ran 1 test in 2.348s
2022-09-27T15:48:15.2735415Z 
2022-09-27T15:48:15.2735519Z OK
2022-09-27T15:48:15.2735700Z 
2022-09-27T15:48:15.2735876Z Generating XML reports...
2022-09-27T15:48:15.2780797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154812.xml
2022-09-27T15:48:16.6233572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:16.6234206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:16.6245426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:16.9819955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:16.9820779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:16.9833442Z 
2022-09-27T15:48:16.9833779Z Running tests...
2022-09-27T15:48:16.9834407Z ----------------------------------------------------------------------
2022-09-27T15:48:17.2228326Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16611
2022-09-27T15:48:17.2255233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16612
2022-09-27T15:48:17.2283044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16613
2022-09-27T15:48:18.2616288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:18.2617247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:18.2645613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:18.2646245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:18.2837572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:18.2838075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:18.2976384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:18.2976992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:18.3004935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:18.3005537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:18.3177494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:18.3178123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:18.6385617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:18.6486283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:18.6486913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:18.6487814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:18.6488386Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:18.6489186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:18.6494775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:18.6495293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:18.6496165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:18.6700502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:18.6802372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:18.6803103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:18.6803795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:18.6804330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:18.6805037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:18.6806953Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6807656Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6907318Z STAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6912074Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6913027Z STAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6913497Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6920162Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6920664Z 
2022-09-27T15:48:18.6925528Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6929961Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6930739Z STAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6931446Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6938707Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6939172Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6942254Z STAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:18.6948194Z STAGE:2022-09-27 15:48:18 16612:16612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6948952Z STAGE:2022-09-27 15:48:18 16611:16611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:18.6949720Z STAGE:2022-09-27 15:48:18 16613:16613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:19.1331401Z ok (2.149s)
2022-09-27T15:48:19.1331651Z 
2022-09-27T15:48:19.1332165Z ----------------------------------------------------------------------
2022-09-27T15:48:19.1332541Z Ran 1 test in 2.150s
2022-09-27T15:48:19.1332661Z 
2022-09-27T15:48:19.1332723Z OK
2022-09-27T15:48:19.1332814Z 
2022-09-27T15:48:19.1332908Z Generating XML reports...
2022-09-27T15:48:19.1378279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154816.xml
2022-09-27T15:48:20.4556654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:20.4557078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:20.4564828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:20.4565389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:20.7929531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:20.7939285Z 
2022-09-27T15:48:20.7939613Z Running tests...
2022-09-27T15:48:20.7940210Z ----------------------------------------------------------------------
2022-09-27T15:48:21.0406082Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16676
2022-09-27T15:48:21.0433367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16677
2022-09-27T15:48:21.0460314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16678
2022-09-27T15:48:22.1136688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:22.1137382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:22.1445325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:22.1445914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:22.1760501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:22.1761149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:22.1824402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:22.1825357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:22.2109281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:22.2109939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:22.2174917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:22.2175559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:22.5456849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:22.5552525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:22.5553262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:22.5553930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:22.5554465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:22.5558230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:22.5660248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:22.5661039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:22.6567748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:22.6772448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:22.6872315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:22.6872727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:22.6873350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:22.6873967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:22.6874538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:22.6878482Z STAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.6978468Z STAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.6978968Z STAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.6984232Z STAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.6985045Z STAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.6993099Z STAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.6994016Z STAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.6994372Z 
2022-09-27T15:48:22.6998446Z STAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.7002605Z STAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.7007841Z STAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.7011714Z STAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.7012039Z 
2022-09-27T15:48:22.7019063Z STAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.7024600Z STAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:22.7029466Z STAGE:2022-09-27 15:48:22 16678:16678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.7030473Z STAGE:2022-09-27 15:48:22 16676:16676 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:22 16677:16677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:22.7030767Z 
2022-09-27T15:48:23.1514770Z ok (2.357s)
2022-09-27T15:48:23.1515010Z 
2022-09-27T15:48:23.1515526Z ----------------------------------------------------------------------
2022-09-27T15:48:23.1515859Z Ran 1 test in 2.357s
2022-09-27T15:48:23.1515975Z 
2022-09-27T15:48:23.1516023Z OK
2022-09-27T15:48:23.1516115Z 
2022-09-27T15:48:23.1516201Z Generating XML reports...
2022-09-27T15:48:23.1561781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154820.xml
2022-09-27T15:48:24.4646740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:24.4654645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:24.4655525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:24.4656147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:24.8028548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:24.8038838Z 
2022-09-27T15:48:24.8039291Z Running tests...
2022-09-27T15:48:24.8039709Z ----------------------------------------------------------------------
2022-09-27T15:48:25.0407299Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16741
2022-09-27T15:48:25.0433004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16742
2022-09-27T15:48:25.0462632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16743
2022-09-27T15:48:26.1206800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:26.1207408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:26.1456765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:26.1457456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:26.1531892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:26.1532548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:26.1786539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:26.1933179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:26.1933933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:26.1934273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:26.2251042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:26.2251430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:26.5662108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:26.5763581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:26.5764029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:26.5764641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:26.5765174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:26.5765707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:26.5870032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:26.6773504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:26.6778026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:26.6980777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:26.7082682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:26.7083411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:26.7084303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:26.7084847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:26.7085359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:26.7088016Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7088699Z STAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7089151Z STAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7093627Z STAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7094122Z 
2022-09-27T15:48:26.7094380Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7101226Z STAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7102032Z STAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7108334Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7113226Z STAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7113968Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7114658Z STAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7120465Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7121016Z STAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7126573Z STAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:26.7131061Z STAGE:2022-09-27 15:48:26 16741:16741 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7131813Z STAGE:2022-09-27 15:48:26 16742:16742 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:26 16743:16743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:26.7132185Z 
2022-09-27T15:48:27.1516554Z ok (2.347s)
2022-09-27T15:48:27.1516828Z 
2022-09-27T15:48:27.1517361Z ----------------------------------------------------------------------
2022-09-27T15:48:27.1517788Z Ran 1 test in 2.348s
2022-09-27T15:48:27.1517911Z 
2022-09-27T15:48:27.1517973Z OK
2022-09-27T15:48:27.1518066Z 
2022-09-27T15:48:27.1518146Z Generating XML reports...
2022-09-27T15:48:27.1562299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154824.xml
2022-09-27T15:48:28.4761396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:28.4761791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:28.4768726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:28.4769102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:28.8172655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:28.8181540Z 
2022-09-27T15:48:28.8181677Z Running tests...
2022-09-27T15:48:28.8182287Z ----------------------------------------------------------------------
2022-09-27T15:48:29.0525103Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16806
2022-09-27T15:48:29.0551618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16807
2022-09-27T15:48:29.0578998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16808
2022-09-27T15:48:30.1211554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:30.1212260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:30.1283259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:30.1283814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:30.1623201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:30.1623621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:30.1632750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:30.1633309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:30.1706591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:30.1707095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:30.2036015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:30.2036678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:30.5226248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:30.5327162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:30.5327702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:30.5328622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:30.5329171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:30.5329777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:30.5335019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:30.5335849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:30.5336309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:30.5336674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:30.5342497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:30.5440618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:30.5441301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:30.5444858Z STAGE:2022-09-27 15:48:30 16807:16807 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:30.5445397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:30.5447972Z STAGE:2022-09-27 15:48:30 16808:16808 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:30.5453486Z STAGE:2022-09-27 15:48:30 16808:16808 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:30.5454247Z STAGE:2022-09-27 15:48:30 16807:16807 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:30.5460221Z STAGE:2022-09-27 15:48:30 16808:16808 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:30.5467064Z STAGE:2022-09-27 15:48:30 16807:16807 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:30.5471616Z STAGE:2022-09-27 15:48:30 16807:16807 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:30.5472335Z STAGE:2022-09-27 15:48:30 16808:16808 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:30.5539535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:30.9627227Z ok (2.144s)
2022-09-27T15:48:30.9627467Z 
2022-09-27T15:48:30.9627988Z ----------------------------------------------------------------------
2022-09-27T15:48:30.9628387Z Ran 1 test in 2.144s
2022-09-27T15:48:30.9628505Z 
2022-09-27T15:48:30.9628565Z OK
2022-09-27T15:48:30.9628659Z 
2022-09-27T15:48:30.9628751Z Generating XML reports...
2022-09-27T15:48:30.9673931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154828.xml
2022-09-27T15:48:32.3059635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:32.3060048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:32.3068023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:32.3068575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:32.6634738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:32.6644068Z 
2022-09-27T15:48:32.6644204Z Running tests...
2022-09-27T15:48:32.6644770Z ----------------------------------------------------------------------
2022-09-27T15:48:32.9041106Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16867
2022-09-27T15:48:32.9067949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16868
2022-09-27T15:48:32.9095446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16869
2022-09-27T15:48:33.9896978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:33.9897369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:33.9958370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:33.9958855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:34.0245077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:34.0245489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:34.0308016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:34.0308674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:34.0357209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:34.0357814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:34.0672279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:34.0672885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:34.3939897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:34.3940467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:34.3940847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:34.3941449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:34.3941987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:34.3942509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:34.4045384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:34.4951149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:34.4952477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:34.4953079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:34.5157560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:34.5157961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:34.5158721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:34.5159267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:34.5159923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:34.5161032Z STAGE:2022-09-27 15:48:34 16868:16868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:34.5161811Z STAGE:2022-09-27 15:48:34 16869:16869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:34.5168355Z STAGE:2022-09-27 15:48:34 16869:16869 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:34 16868:16868 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:34.5168720Z 
2022-09-27T15:48:34.5175517Z STAGE:2022-09-27 15:48:34 16869:16869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:34.5181147Z STAGE:2022-09-27 15:48:34 16868:16868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:34.5186369Z STAGE:2022-09-27 15:48:34 16868:16868 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:34.5186871Z STAGE:2022-09-27 15:48:34 16869:16869 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:34.9144687Z ok (2.250s)
2022-09-27T15:48:34.9145033Z 
2022-09-27T15:48:34.9145436Z ----------------------------------------------------------------------
2022-09-27T15:48:34.9145709Z Ran 1 test in 2.250s
2022-09-27T15:48:34.9145827Z 
2022-09-27T15:48:34.9145891Z OK
2022-09-27T15:48:34.9145970Z 
2022-09-27T15:48:34.9146080Z Generating XML reports...
2022-09-27T15:48:34.9190703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154832.xml
2022-09-27T15:48:36.2401284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:36.2401674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:36.2409539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:36.2409910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:36.5816441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:36.5825869Z 
2022-09-27T15:48:36.5826319Z Running tests...
2022-09-27T15:48:36.5826725Z ----------------------------------------------------------------------
2022-09-27T15:48:36.8195120Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16928
2022-09-27T15:48:36.8221232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16929
2022-09-27T15:48:36.8248199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16930
2022-09-27T15:48:37.8647534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:37.8648195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:37.8859716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:37.8860383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:37.8972706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:37.8973363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:37.9187405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:37.9188095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:37.9227616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:37.9228251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:37.9552426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:37.9553096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:38.3067842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:38.3168452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:38.3169106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:38.3179476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:38.3180619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:38.3181314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:38.3181694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:38.3182052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:38.3182397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:38.3182758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:38.3285999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:38.3286576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:38.3287510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:38.3288344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:38.3289814Z STAGE:2022-09-27 15:48:38 16929:16929 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:38.3291484Z STAGE:2022-09-27 15:48:38 16930:16930 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:38.3297231Z STAGE:2022-09-27 15:48:38 16930:16930 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:38.3297710Z STAGE:2022-09-27 15:48:38 16929:16929 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:38.3305169Z STAGE:2022-09-27 15:48:38 16930:16930 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:38.3310114Z STAGE:2022-09-27 15:48:38 16929:16929 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:38.3314508Z STAGE:2022-09-27 15:48:38 16929:16929 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:38.3314970Z STAGE:2022-09-27 15:48:38 16930:16930 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:38.3384090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:38.8301480Z ok (2.247s)
2022-09-27T15:48:38.8301709Z 
2022-09-27T15:48:38.8302241Z ----------------------------------------------------------------------
2022-09-27T15:48:38.8302706Z Ran 1 test in 2.248s
2022-09-27T15:48:38.8302870Z 
2022-09-27T15:48:38.8302933Z OK
2022-09-27T15:48:38.8303029Z 
2022-09-27T15:48:38.8303122Z Generating XML reports...
2022-09-27T15:48:38.8348501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154836.xml
2022-09-27T15:48:40.1389517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:40.1389899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:40.1397667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:40.1398019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:40.4764813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:40.4774139Z 
2022-09-27T15:48:40.4774294Z Running tests...
2022-09-27T15:48:40.4775143Z ----------------------------------------------------------------------
2022-09-27T15:48:40.7150972Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16989
2022-09-27T15:48:40.7178150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16990
2022-09-27T15:48:40.7205810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16991
2022-09-27T15:48:41.7724129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:41.7724533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:41.7737244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:41.7737821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:41.7740075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:41.7740680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:41.8089578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:41.8089958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:41.8097789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:41.8098381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:41.8106818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:41.8107163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:42.1426445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:42.1526957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:42.1527498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:42.1528159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:42.1528885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:42.1529741Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:42.1534779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:42.1535376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:42.1536623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:48:42.1537012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:42.1642507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:48:42.1643224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:48:42.1643849Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:42.1644371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:42.1647022Z STAGE:2022-09-27 15:48:42 16990:16990 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:42.1740229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:48:42.1746953Z STAGE:2022-09-27 15:48:42 16991:16991 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:42.1753390Z STAGE:2022-09-27 15:48:42 16990:16990 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:42 16991:16991 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:42.1754048Z 
2022-09-27T15:48:42.1760170Z STAGE:2022-09-27 15:48:42 16991:16991 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:42.1765892Z STAGE:2022-09-27 15:48:42 16990:16990 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:42.1770929Z STAGE:2022-09-27 15:48:42 16991:16991 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:42 16990:16990 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:42.1771238Z 
2022-09-27T15:48:42.6255777Z ok (2.148s)
2022-09-27T15:48:42.6255991Z 
2022-09-27T15:48:42.6256338Z ----------------------------------------------------------------------
2022-09-27T15:48:42.6256581Z Ran 1 test in 2.148s
2022-09-27T15:48:42.6256699Z 
2022-09-27T15:48:42.6256775Z OK
2022-09-27T15:48:42.6256871Z 
2022-09-27T15:48:42.6256967Z Generating XML reports...
2022-09-27T15:48:42.6301212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154840.xml
2022-09-27T15:48:43.9306206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:43.9306611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:43.9314288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:43.9314655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:44.2681049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:44.2690274Z 
2022-09-27T15:48:44.2690399Z Running tests...
2022-09-27T15:48:44.2690982Z ----------------------------------------------------------------------
2022-09-27T15:48:44.5041500Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17050
2022-09-27T15:48:44.5067927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17051
2022-09-27T15:48:44.5094786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17052
2022-09-27T15:48:45.5338229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:45.5338904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:45.5646892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:45.5647540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:45.5828096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:45.5828767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:45.5968557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:45.5969138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:45.6165623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:45.6166256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:45.6308067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:45.6308738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:45.9477893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:45.9578297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:45.9579065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:45.9579698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:45.9580231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:45.9580761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:45.9684121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:45.9688101Z STAGE:2022-09-27 15:48:45 17052:17052 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0587754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:46.0588498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:46.0592871Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0594022Z STAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0600210Z STAGE:2022-09-27 15:48:46 17052:17052 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0600707Z 
2022-09-27T15:48:46.0601118Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0608209Z STAGE:2022-09-27 15:48:46 17052:17052 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0608714Z STAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0614673Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0620002Z STAGE:2022-09-27 15:48:46 17052:17052 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0620835Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0621893Z STAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0625412Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0625997Z STAGE:2022-09-27 15:48:46 17052:17052 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0631867Z STAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:46.0637085Z STAGE:2022-09-27 15:48:46 17050:17050 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0638044Z STAGE:2022-09-27 15:48:46 17052:17052 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:46 17051:17051 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:46.0638337Z 
2022-09-27T15:48:46.4143547Z ok (2.145s)
2022-09-27T15:48:46.4143746Z 
2022-09-27T15:48:46.4144782Z ----------------------------------------------------------------------
2022-09-27T15:48:46.4145297Z Ran 1 test in 2.145s
2022-09-27T15:48:46.4145522Z 
2022-09-27T15:48:46.4145615Z OK
2022-09-27T15:48:46.4145709Z 
2022-09-27T15:48:46.4145807Z Generating XML reports...
2022-09-27T15:48:46.4190505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154844.xml
2022-09-27T15:48:47.7242560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:47.7243237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:47.7251372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:47.7251995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:48.0660024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:48.0670396Z 
2022-09-27T15:48:48.0670752Z Running tests...
2022-09-27T15:48:48.0671380Z ----------------------------------------------------------------------
2022-09-27T15:48:48.3042182Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17106
2022-09-27T15:48:48.3068029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17107
2022-09-27T15:48:48.3095150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17108
2022-09-27T15:48:49.3778947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:49.3779617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:49.4072600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:49.4073191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:49.4101366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:49.4101925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:49.4326336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:49.4327008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:49.4397692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:49.4398248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:49.4654047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:49.4654665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:49.7865811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:49.7966596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:49.7967344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:49.7968162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:49.7969166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:49.7970038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:49.8073670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:49.8077867Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.8981307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:49.8981918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:49.8986033Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.8987279Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.8992257Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.8993127Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9000604Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9002113Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9005955Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9018833Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9023418Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9024316Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9025089Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9029829Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9031020Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9037919Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:49.9042815Z STAGE:2022-09-27 15:48:49 17106:17106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9043585Z STAGE:2022-09-27 15:48:49 17107:17107 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:49.9044374Z STAGE:2022-09-27 15:48:49 17108:17108 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:50.3147657Z ok (2.247s)
2022-09-27T15:48:50.3147891Z 
2022-09-27T15:48:50.3148378Z ----------------------------------------------------------------------
2022-09-27T15:48:50.3148828Z Ran 1 test in 2.248s
2022-09-27T15:48:50.3148980Z 
2022-09-27T15:48:50.3149032Z OK
2022-09-27T15:48:50.3149126Z 
2022-09-27T15:48:50.3149220Z Generating XML reports...
2022-09-27T15:48:50.3194167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154848.xml
2022-09-27T15:48:51.6250114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:51.6250492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:51.6258265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:51.6258768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:51.9631280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:51.9640704Z 
2022-09-27T15:48:51.9640843Z Running tests...
2022-09-27T15:48:51.9641656Z ----------------------------------------------------------------------
2022-09-27T15:48:51.9660037Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T15:48:51.9660803Z 
2022-09-27T15:48:51.9661446Z ----------------------------------------------------------------------
2022-09-27T15:48:51.9661950Z Ran 1 test in 0.002s
2022-09-27T15:48:51.9662149Z 
2022-09-27T15:48:51.9662237Z OK (skipped=1)
2022-09-27T15:48:51.9662347Z 
2022-09-27T15:48:51.9662436Z Generating XML reports...
2022-09-27T15:48:51.9692255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154851.xml
2022-09-27T15:48:53.1569118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:53.1569507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:53.1577017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:53.1577474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:53.5019211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:53.5028929Z 
2022-09-27T15:48:53.5029061Z Running tests...
2022-09-27T15:48:53.5030130Z ----------------------------------------------------------------------
2022-09-27T15:48:53.7396392Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17172
2022-09-27T15:48:53.7423130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17173
2022-09-27T15:48:53.7450576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17174
2022-09-27T15:48:54.8013464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:54.8014123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:54.8036459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:54.8037133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:54.8246111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:54.8246623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:54.8359777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:54.8360517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:54.8392646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:54.8393195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:54.8582343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:54.8582974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:55.1963926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:55.2065331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:48:55.2065991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:55.2066956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:55.2067557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:55.2068139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:48:55.2074829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:55.2075407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:48:55.2078947Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.2079267Z 
2022-09-27T15:48:55.2083927Z STAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.2091434Z STAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3074837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:55.3077786Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3083485Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3084208Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3091249Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3097950Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3102586Z STAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3103124Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3103587Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3109892Z STAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3110549Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3114838Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:48:55.3120210Z STAGE:2022-09-27 15:48:55 17172:17172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3121570Z STAGE:2022-09-27 15:48:55 17173:17173 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:48:55 17174:17174 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:48:55.3121882Z 
2022-09-27T15:48:55.7500420Z ok (2.247s)
2022-09-27T15:48:55.7500647Z 
2022-09-27T15:48:55.7501140Z ----------------------------------------------------------------------
2022-09-27T15:48:55.7501580Z Ran 1 test in 2.247s
2022-09-27T15:48:55.7501724Z 
2022-09-27T15:48:55.7501787Z OK
2022-09-27T15:48:55.7501894Z 
2022-09-27T15:48:55.7501987Z Generating XML reports...
2022-09-27T15:48:55.7546801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154853.xml
2022-09-27T15:48:57.0829180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:57.0829872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:57.0838654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:57.0839051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:57.4420516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:57.4431752Z 
2022-09-27T15:48:57.4431903Z Running tests...
2022-09-27T15:48:57.4432538Z ----------------------------------------------------------------------
2022-09-27T15:48:57.4461388Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-09-27T15:48:57.4461766Z 
2022-09-27T15:48:57.4462505Z ----------------------------------------------------------------------
2022-09-27T15:48:57.4463008Z Ran 1 test in 0.003s
2022-09-27T15:48:57.4463174Z 
2022-09-27T15:48:57.4463253Z OK (skipped=1)
2022-09-27T15:48:57.4463362Z 
2022-09-27T15:48:57.4463448Z Generating XML reports...
2022-09-27T15:48:57.4493308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154857.xml
2022-09-27T15:48:58.6378437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:58.6379046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:58.6387117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:58.6387589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:58.9775274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:58.9785047Z 
2022-09-27T15:48:58.9785186Z Running tests...
2022-09-27T15:48:58.9786292Z ----------------------------------------------------------------------
2022-09-27T15:48:59.2168392Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17238
2022-09-27T15:48:59.2194648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17239
2022-09-27T15:48:59.2221381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17240
2022-09-27T15:49:00.2760696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:00.2761347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:00.2911256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:00.2911853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:00.3098864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:00.3099259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:00.3215259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:00.3215746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:00.3249405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:00.3249867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:00.3537627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:00.3538286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:00.6913865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:00.6914344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:00.6914727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:00.6915366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:00.6915988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:00.6916522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:00.6922549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:00.6923452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:00.6924542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:00.6926558Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6928839Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6931042Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6936530Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6937316Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6938117Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6944518Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6945579Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6949530Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6955013Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6955670Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6956121Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6960190Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6961470Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6967180Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:00.6972157Z STAGE:2022-09-27 15:49:00 17239:17239 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6973143Z STAGE:2022-09-27 15:49:00 17238:17238 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:00.6973946Z STAGE:2022-09-27 15:49:00 17240:17240 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:01.1269743Z ok (2.148s)
2022-09-27T15:49:01.1269968Z 
2022-09-27T15:49:01.1270518Z ----------------------------------------------------------------------
2022-09-27T15:49:01.1270942Z Ran 1 test in 2.148s
2022-09-27T15:49:01.1271135Z 
2022-09-27T15:49:01.1271247Z OK
2022-09-27T15:49:01.1271413Z 
2022-09-27T15:49:01.1271570Z Generating XML reports...
2022-09-27T15:49:01.1316230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154858.xml
2022-09-27T15:49:02.4702801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:02.4703149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:02.4711414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:02.4711782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:02.8109978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:02.8119366Z 
2022-09-27T15:49:02.8119638Z Running tests...
2022-09-27T15:49:02.8120017Z ----------------------------------------------------------------------
2022-09-27T15:49:02.8140157Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T15:49:02.8140441Z 
2022-09-27T15:49:02.8140755Z ----------------------------------------------------------------------
2022-09-27T15:49:02.8141006Z Ran 1 test in 0.002s
2022-09-27T15:49:02.8141123Z 
2022-09-27T15:49:02.8141196Z OK (skipped=1)
2022-09-27T15:49:02.8141312Z 
2022-09-27T15:49:02.8141436Z Generating XML reports...
2022-09-27T15:49:02.8171413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154902.xml
2022-09-27T15:49:04.0293898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:04.0294561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:04.0303063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:04.0303728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:04.3694443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:04.3703435Z 
2022-09-27T15:49:04.3703561Z Running tests...
2022-09-27T15:49:04.3704451Z ----------------------------------------------------------------------
2022-09-27T15:49:04.3723118Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T15:49:04.3723475Z 
2022-09-27T15:49:04.3723934Z ----------------------------------------------------------------------
2022-09-27T15:49:04.3724223Z Ran 1 test in 0.002s
2022-09-27T15:49:04.3724337Z 
2022-09-27T15:49:04.3724414Z OK (skipped=1)
2022-09-27T15:49:04.3724528Z 
2022-09-27T15:49:04.3724616Z Generating XML reports...
2022-09-27T15:49:04.3755484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154904.xml
2022-09-27T15:49:05.5612695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:05.5613355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:05.5621594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:05.5622240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:05.9001239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:05.9010783Z 
2022-09-27T15:49:05.9010938Z Running tests...
2022-09-27T15:49:05.9011408Z ----------------------------------------------------------------------
2022-09-27T15:49:06.1357131Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17314
2022-09-27T15:49:06.1383400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17315
2022-09-27T15:49:06.1410393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17316
2022-09-27T15:49:07.2613418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:07.2613806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:07.2735748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:07.2736093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:07.2956662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:07.2957331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:07.3014668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:07.3015297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:07.3093840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:07.3094462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:07.3359843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:07.3360505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:07.6577344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:07.6678738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:07.6679366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:07.6680130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:07.6681167Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:07.6681785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:07.6685740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:07.6686769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:07.6687790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:07.6691240Z STAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6692570Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6693014Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6700812Z STAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6701541Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6702326Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6711025Z STAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6711726Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6719532Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6726184Z STAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6726843Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6727295Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6734112Z STAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6734632Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6740560Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:07.6749797Z STAGE:2022-09-27 15:49:07 17314:17314 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6750843Z STAGE:2022-09-27 15:49:07 17315:17315 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:49:07 17316:17316 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:07.6751310Z 
2022-09-27T15:49:08.0458779Z ok (2.144s)
2022-09-27T15:49:08.0459082Z 
2022-09-27T15:49:08.0459556Z ----------------------------------------------------------------------
2022-09-27T15:49:08.0460006Z Ran 1 test in 2.145s
2022-09-27T15:49:08.0460134Z 
2022-09-27T15:49:08.0460195Z OK
2022-09-27T15:49:08.0460273Z 
2022-09-27T15:49:08.0460367Z Generating XML reports...
2022-09-27T15:49:08.0507045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154905.xml
2022-09-27T15:49:09.3814103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:09.3814480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:09.3822252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:09.3822620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:09.7199967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:09.7209493Z 
2022-09-27T15:49:09.7209660Z Running tests...
2022-09-27T15:49:09.7210094Z ----------------------------------------------------------------------
2022-09-27T15:49:09.9552619Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17370
2022-09-27T15:49:09.9579383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17371
2022-09-27T15:49:09.9606304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17372
2022-09-27T15:49:11.0624327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:11.0624746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:11.0982925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:11.0983401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:11.1081804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:11.1082402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:11.1152247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:11.1152588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:11.1427906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:11.1428467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:11.1508631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:11.1509194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:11.4878664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:11.4979363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:11.4979832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:11.4980477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:11.4981010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:11.4981534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:11.5087168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:11.5087939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:11.5092047Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.5092957Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.5988891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:11.5994152Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.5999683Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6000624Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6001334Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6010157Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6010716Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6011353Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6015172Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6016049Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6016729Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6020853Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6021669Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6022434Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:11.6026961Z STAGE:2022-09-27 15:49:11 17370:17370 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6027701Z STAGE:2022-09-27 15:49:11 17371:17371 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.6028408Z STAGE:2022-09-27 15:49:11 17372:17372 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:11.9657446Z ok (2.244s)
2022-09-27T15:49:11.9657680Z 
2022-09-27T15:49:11.9658129Z ----------------------------------------------------------------------
2022-09-27T15:49:11.9658540Z Ran 1 test in 2.245s
2022-09-27T15:49:11.9658718Z 
2022-09-27T15:49:11.9658796Z OK
2022-09-27T15:49:11.9658945Z 
2022-09-27T15:49:11.9659106Z Generating XML reports...
2022-09-27T15:49:11.9706076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154909.xml
2022-09-27T15:49:13.2649222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:13.2649590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:13.2657680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:13.2658074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:13.6028994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:13.6038333Z 
2022-09-27T15:49:13.6038431Z Running tests...
2022-09-27T15:49:13.6039020Z ----------------------------------------------------------------------
2022-09-27T15:49:13.8386137Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17426
2022-09-27T15:49:13.8413038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17427
2022-09-27T15:49:13.8439801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17428
2022-09-27T15:49:14.9316580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:14.9317199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:14.9385060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:14.9385753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:14.9450947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:14.9451792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:14.9670371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:14.9671341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:14.9751119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:14.9751740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:14.9806048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:14.9806461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:15.3030160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:15.3081950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:15.3082582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:15.3083212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:15.3083758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:15.3130901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:15.3188671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:15.3189409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:15.4138904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:15.7488883Z ok (2.145s)
2022-09-27T15:49:15.7489096Z 
2022-09-27T15:49:15.7489609Z ----------------------------------------------------------------------
2022-09-27T15:49:15.7490060Z Ran 1 test in 2.145s
2022-09-27T15:49:15.7490186Z 
2022-09-27T15:49:15.7490288Z OK
2022-09-27T15:49:15.7490414Z 
2022-09-27T15:49:15.7490500Z Generating XML reports...
2022-09-27T15:49:15.7536923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154913.xml
2022-09-27T15:49:17.0742316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:17.0742680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:17.0750703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:17.0751206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:17.4122800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:17.4131623Z 
2022-09-27T15:49:17.4131901Z Running tests...
2022-09-27T15:49:17.4132526Z ----------------------------------------------------------------------
2022-09-27T15:49:17.6465681Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17479
2022-09-27T15:49:17.6492748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17480
2022-09-27T15:49:17.6519931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17481
2022-09-27T15:49:18.7131258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.7131891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.7475044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.7476001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:18.7763142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.7764061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.7930130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.7930638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.8099566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.8100187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:18.8267498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.8267902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:19.1725330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:19.1726090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:19.1726630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:19.1727251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:19.1727771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:19.1728297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:19.1831491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:19.1835605Z STAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2735868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:19.2736469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:19.2740805Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2741830Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2748780Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2749431Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2749962Z STAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2759853Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2763721Z STAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2764755Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2769742Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2770495Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2771200Z STAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2776531Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2777714Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2779769Z STAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:19.2781618Z STAGE:2022-09-27 15:49:19 17479:17479 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2783447Z STAGE:2022-09-27 15:49:19 17481:17481 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:49:19 17480:17480 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:19.2783976Z 
2022-09-27T15:49:19.6569762Z ok (2.243s)
2022-09-27T15:49:19.6569965Z 
2022-09-27T15:49:19.6570293Z ----------------------------------------------------------------------
2022-09-27T15:49:19.6570547Z Ran 1 test in 2.244s
2022-09-27T15:49:19.6570667Z 
2022-09-27T15:49:19.6570716Z OK
2022-09-27T15:49:19.6570808Z 
2022-09-27T15:49:19.6570899Z Generating XML reports...
2022-09-27T15:49:19.6615651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154917.xml
2022-09-27T15:49:20.9630655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:20.9631301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:20.9639084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:20.9639739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:21.3038133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:21.3048232Z 
2022-09-27T15:49:21.3048333Z Running tests...
2022-09-27T15:49:21.3049281Z ----------------------------------------------------------------------
2022-09-27T15:49:21.3066117Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-09-27T15:49:21.3066476Z 
2022-09-27T15:49:21.3066920Z ----------------------------------------------------------------------
2022-09-27T15:49:21.3067271Z Ran 1 test in 0.002s
2022-09-27T15:49:21.3067387Z 
2022-09-27T15:49:21.3067459Z OK (skipped=1)
2022-09-27T15:49:21.3067567Z 
2022-09-27T15:49:21.3067646Z Generating XML reports...
2022-09-27T15:49:21.3096896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154921.xml
2022-09-27T15:49:22.4950894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:22.4951280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:22.4958797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:22.4959181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:22.8352548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:22.8361879Z 
2022-09-27T15:49:22.8361986Z Running tests...
2022-09-27T15:49:22.8362412Z ----------------------------------------------------------------------
2022-09-27T15:49:22.8378973Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-09-27T15:49:22.8379318Z 
2022-09-27T15:49:22.8379655Z ----------------------------------------------------------------------
2022-09-27T15:49:22.8380143Z Ran 1 test in 0.002s
2022-09-27T15:49:22.8380323Z 
2022-09-27T15:49:22.8380398Z OK (skipped=1)
2022-09-27T15:49:22.8380494Z 
2022-09-27T15:49:22.8380578Z Generating XML reports...
2022-09-27T15:49:22.8410470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154922.xml
2022-09-27T15:49:24.0649601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:24.0649995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:24.0657347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:24.0657744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:24.4058458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:24.4067537Z 
2022-09-27T15:49:24.4067657Z Running tests...
2022-09-27T15:49:24.4068040Z ----------------------------------------------------------------------
2022-09-27T15:49:24.6412969Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17555
2022-09-27T15:49:24.6439317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17556
2022-09-27T15:49:24.6466940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17557
2022-09-27T15:49:25.7293206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:25.7293621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:25.7608563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:25.7609188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:25.7685381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:25.7685977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:25.7751855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:25.7752462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:25.8030749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:25.8031415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:25.8101009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:25.8101415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:26.1446244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:26.1546514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:26.1547240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:26.1548007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:26.1548525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:26.1549066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:26.1653025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:26.2557178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:26.2557841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:26.2864982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:49:26.2865687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:49:26.2866152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:49:26.2866938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:26.2867476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:26.2868468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:26.2869647Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2869939Z 
2022-09-27T15:49:26.2870259Z STAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2876870Z STAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2877987Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2878431Z 
2022-09-27T15:49:26.2887434Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2887976Z 
2022-09-27T15:49:26.2888446Z STAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2891271Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2892015Z STAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2892462Z STAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2896836Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2897411Z STAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2898494Z STAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:26.2903014Z STAGE:2022-09-27 15:49:26 17555:17555 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:49:26 17556:17556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.2903470Z 
2022-09-27T15:49:26.2904088Z STAGE:2022-09-27 15:49:26 17557:17557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:26.6519282Z ok (2.245s)
2022-09-27T15:49:26.6519500Z 
2022-09-27T15:49:26.6520043Z ----------------------------------------------------------------------
2022-09-27T15:49:26.6520454Z Ran 1 test in 2.245s
2022-09-27T15:49:26.6520583Z 
2022-09-27T15:49:26.6520631Z OK
2022-09-27T15:49:26.6520724Z 
2022-09-27T15:49:26.6520817Z Generating XML reports...
2022-09-27T15:49:26.6566097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154924.xml
2022-09-27T15:49:27.9556087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:27.9556736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:27.9564502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:27.9565147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:28.2996045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:28.3006324Z 
2022-09-27T15:49:28.3006665Z Running tests...
2022-09-27T15:49:28.3007284Z ----------------------------------------------------------------------
2022-09-27T15:49:28.5366581Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17620
2022-09-27T15:49:28.5392753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17621
2022-09-27T15:49:28.5419774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17622
2022-09-27T15:49:29.6157745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:29.6158394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:29.6210593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:29.6210980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:29.6285873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:29.6286403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:29.6508928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:29.6509407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:29.6570313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:29.6570838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:29.6638301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:29.6638813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:30.0069990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:30.0070589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:30.0071152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:30.0072081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:30.0072897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:30.0073671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:30.0175401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:30.1080822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:30.1081395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:30.1082425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:49:30.1286651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:49:30.1287386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:49:30.1288234Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:30.1288778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:30.1292538Z STAGE:2022-09-27 15:49:30 17621:17621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:30.1387399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:49:30.1391751Z STAGE:2022-09-27 15:49:30 17622:17622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:30.1396761Z STAGE:2022-09-27 15:49:30 17622:17622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:30.1397233Z STAGE:2022-09-27 15:49:30 17621:17621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:30.1407363Z STAGE:2022-09-27 15:49:30 17621:17621 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:49:30 17622:17622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:30.1407690Z 
2022-09-27T15:49:30.1411093Z STAGE:2022-09-27 15:49:30 17622:17622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:30.1411850Z STAGE:2022-09-27 15:49:30 17621:17621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:30.5473360Z ok (2.246s)
2022-09-27T15:49:30.5473579Z 
2022-09-27T15:49:30.5474009Z ----------------------------------------------------------------------
2022-09-27T15:49:30.5474314Z Ran 1 test in 2.247s
2022-09-27T15:49:30.5474477Z 
2022-09-27T15:49:30.5474557Z OK
2022-09-27T15:49:30.5474664Z 
2022-09-27T15:49:30.5474746Z Generating XML reports...
2022-09-27T15:49:30.5521765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154928.xml
2022-09-27T15:49:31.8845585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:31.8845936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:31.8853588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:31.8854070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:32.2218567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:32.2228041Z 
2022-09-27T15:49:32.2228356Z Running tests...
2022-09-27T15:49:32.2229045Z ----------------------------------------------------------------------
2022-09-27T15:49:32.4583228Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17681
2022-09-27T15:49:32.4611023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17682
2022-09-27T15:49:32.4637947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17683
2022-09-27T15:49:33.4948138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.4948704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.5084732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.5085397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.5283946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.5284361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.5428320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.5428710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.5557123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.5557744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.5873935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.5874596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.9242776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:33.9243690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:33.9244077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:33.9244786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:33.9245316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:33.9245825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:33.9252917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:33.9253365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:33.9253723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:34.2686664Z ok (2.045s)
2022-09-27T15:49:34.2686938Z 
2022-09-27T15:49:34.2687463Z ----------------------------------------------------------------------
2022-09-27T15:49:34.2687711Z Ran 1 test in 2.046s
2022-09-27T15:49:34.2687834Z 
2022-09-27T15:49:34.2687894Z OK
2022-09-27T15:49:34.2687984Z 
2022-09-27T15:49:34.2688075Z Generating XML reports...
2022-09-27T15:49:34.2732394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154932.xml
2022-09-27T15:49:35.5428406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:35.5428788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:35.5436391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:35.5436959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:35.8813426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:35.8823140Z 
2022-09-27T15:49:35.8823271Z Running tests...
2022-09-27T15:49:35.8823994Z ----------------------------------------------------------------------
2022-09-27T15:49:36.1171728Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17734
2022-09-27T15:49:36.1197657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17735
2022-09-27T15:49:36.1224394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17736
2022-09-27T15:49:37.1790105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:37.1790749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:37.2084923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:37.2085639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:37.2107303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:37.2107981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:37.2197884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:37.2198258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:37.2429378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:37.2429747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:37.2541807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:37.2542426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:37.6003800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:37.6004403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:37.6004781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:37.6005388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:37.6005920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:37.6006447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:37.6110653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:37.6111220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:37.7012915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:38.0275509Z ok (2.145s)
2022-09-27T15:49:38.0275759Z 
2022-09-27T15:49:38.0276283Z ----------------------------------------------------------------------
2022-09-27T15:49:38.0276641Z Ran 1 test in 2.145s
2022-09-27T15:49:38.0276757Z 
2022-09-27T15:49:38.0276820Z OK
2022-09-27T15:49:38.0276913Z 
2022-09-27T15:49:38.0277005Z Generating XML reports...
2022-09-27T15:49:38.0321772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154935.xml
2022-09-27T15:49:39.3489067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:39.3489710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:39.3497922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:39.3498570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:39.6895507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:39.6906012Z 
2022-09-27T15:49:39.6906480Z Running tests...
2022-09-27T15:49:39.6906901Z ----------------------------------------------------------------------
2022-09-27T15:49:39.9225839Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17787
2022-09-27T15:49:39.9250739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17788
2022-09-27T15:49:39.9277410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17789
2022-09-27T15:49:41.0009993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:41.0010603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:41.0356199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:41.0356884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:41.0549468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:41.0550096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:41.0846209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:41.0846869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:41.0909463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:41.0910086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:41.1174360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:41.1174895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:41.4659652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:41.4760202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:41.4760933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:41.4761613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:41.4762135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:41.4762677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:41.4868623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:41.4869018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:41.5774838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:41.9329490Z ok (2.242s)
2022-09-27T15:49:41.9329813Z 
2022-09-27T15:49:41.9330361Z ----------------------------------------------------------------------
2022-09-27T15:49:41.9330661Z Ran 1 test in 2.242s
2022-09-27T15:49:41.9330780Z 
2022-09-27T15:49:41.9330845Z OK
2022-09-27T15:49:41.9330924Z 
2022-09-27T15:49:41.9331020Z Generating XML reports...
2022-09-27T15:49:41.9375610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154939.xml
2022-09-27T15:49:43.2573534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.2573922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.2581293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.2581798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.5950867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:43.5959802Z 
2022-09-27T15:49:43.5959942Z Running tests...
2022-09-27T15:49:43.5960579Z ----------------------------------------------------------------------
2022-09-27T15:49:43.8315812Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17840
2022-09-27T15:49:43.8342758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17841
2022-09-27T15:49:43.8369852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17842
2022-09-27T15:49:44.9186455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:44.9187135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:44.9506829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:44.9507438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:44.9612047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:44.9612656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:44.9701012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:44.9701613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:44.9959044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:44.9960009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:45.0039198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:45.0039868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:45.3502351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:45.3503045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:45.3503426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:45.3504180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:45.3504703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:45.3505330Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:45.3607670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:45.3611095Z STAGE:2022-09-27 15:49:45 17841:17841 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:45.4512168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:45.4512534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:45.4514899Z STAGE:2022-09-27 15:49:45 17840:17840 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:45.4517127Z STAGE:2022-09-27 15:49:45 17842:17842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:45.4540498Z STAGE:2022-09-27 15:49:45 17840:17840 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:45.4541323Z STAGE:2022-09-27 15:49:45 17841:17841 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:45.4542261Z STAGE:2022-09-27 15:49:45 17842:17842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:45.8422515Z ok (2.246s)
2022-09-27T15:49:45.8422783Z 
2022-09-27T15:49:45.8423339Z ----------------------------------------------------------------------
2022-09-27T15:49:45.8423716Z Ran 1 test in 2.246s
2022-09-27T15:49:45.8423944Z 
2022-09-27T15:49:45.8424021Z OK
2022-09-27T15:49:45.8424105Z 
2022-09-27T15:49:45.8424203Z Generating XML reports...
2022-09-27T15:49:45.8470253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154943.xml
2022-09-27T15:49:47.1441566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:47.1441947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:47.1449106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:47.1449707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:47.4832130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:47.4842486Z 
2022-09-27T15:49:47.4842617Z Running tests...
2022-09-27T15:49:47.4843212Z ----------------------------------------------------------------------
2022-09-27T15:49:47.7213464Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17896
2022-09-27T15:49:47.7240859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17897
2022-09-27T15:49:47.7267259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17898
2022-09-27T15:49:48.7881368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:48.7881949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:48.8198419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:48.8198840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:48.8793183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:48.8793612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:48.8877375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:48.8877850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:48.9137623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:48.9138128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:48.9220213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:48.9220667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:49.2417650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:49.2518919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:49.2519798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:49.2520324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:49.2521149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:49.2521836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:49.2527649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:49.2528327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:49.2528688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:49.2532180Z STAGE:2022-09-27 15:49:49 17896:17896 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:49.2532653Z STAGE:2022-09-27 15:49:49 17897:17897 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:49.2535139Z STAGE:2022-09-27 15:49:49 17898:17898 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:49.2553647Z STAGE:2022-09-27 15:49:49 17896:17896 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:49.2554897Z STAGE:2022-09-27 15:49:49 17897:17897 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:49.2555698Z STAGE:2022-09-27 15:49:49 17898:17898 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:49.6319240Z ok (2.147s)
2022-09-27T15:49:49.6319492Z 
2022-09-27T15:49:49.6319925Z ----------------------------------------------------------------------
2022-09-27T15:49:49.6320290Z Ran 1 test in 2.148s
2022-09-27T15:49:49.6320478Z 
2022-09-27T15:49:49.6320574Z OK
2022-09-27T15:49:49.6320720Z 
2022-09-27T15:49:49.6320865Z Generating XML reports...
2022-09-27T15:49:49.6366799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154947.xml
2022-09-27T15:49:50.9506781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:50.9507333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:50.9513895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:50.9514351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:51.2939855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:51.2949180Z 
2022-09-27T15:49:51.2949277Z Running tests...
2022-09-27T15:49:51.2950571Z ----------------------------------------------------------------------
2022-09-27T15:49:51.5316138Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17952
2022-09-27T15:49:51.5342097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17953
2022-09-27T15:49:51.5369125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17954
2022-09-27T15:49:52.6015290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.6015995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.6314443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.6315153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.6348000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.6348628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.6431393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.6432005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.6670351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.6671022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.6791133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.6791791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.9968024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:53.0068322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:49:53.0068910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:53.0069687Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:53.0070665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:53.0071383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:49:53.0175559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:49:53.0178616Z STAGE:2022-09-27 15:49:53 17954:17954 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:53.1080415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:53.1080835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:53.1084133Z STAGE:2022-09-27 15:49:53 17953:17953 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:53.1084849Z STAGE:2022-09-27 15:49:53 17952:17952 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:49:53.1110230Z STAGE:2022-09-27 15:49:53 17954:17954 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:53.1113838Z STAGE:2022-09-27 15:49:53 17952:17952 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:53.1114663Z STAGE:2022-09-27 15:49:53 17953:17953 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:49:53.4420383Z ok (2.147s)
2022-09-27T15:49:53.4420689Z 
2022-09-27T15:49:53.4421219Z ----------------------------------------------------------------------
2022-09-27T15:49:53.4421664Z Ran 1 test in 2.147s
2022-09-27T15:49:53.4421799Z 
2022-09-27T15:49:53.4421867Z OK
2022-09-27T15:49:53.4421960Z 
2022-09-27T15:49:53.4422042Z Generating XML reports...
2022-09-27T15:49:53.4467566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154951.xml
2022-09-27T15:49:54.7460304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:54.7460683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:54.7468530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:54.7469074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:55.0857040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:55.0866680Z 
2022-09-27T15:49:55.0866831Z Running tests...
2022-09-27T15:49:55.0867445Z ----------------------------------------------------------------------
2022-09-27T15:49:55.0884184Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-09-27T15:49:55.0884506Z 
2022-09-27T15:49:55.0884815Z ----------------------------------------------------------------------
2022-09-27T15:49:55.0885120Z Ran 1 test in 0.002s
2022-09-27T15:49:55.0885248Z 
2022-09-27T15:49:55.0885322Z OK (skipped=1)
2022-09-27T15:49:55.0885430Z 
2022-09-27T15:49:55.0885503Z Generating XML reports...
2022-09-27T15:49:55.0914782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154955.xml
2022-09-27T15:49:56.2892591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:56.2892977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:56.2900386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:56.2900894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:56.6271645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:56.6281249Z 
2022-09-27T15:49:56.6281390Z Running tests...
2022-09-27T15:49:56.6282045Z ----------------------------------------------------------------------
2022-09-27T15:49:56.6297743Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-09-27T15:49:56.6298171Z 
2022-09-27T15:49:56.6298677Z ----------------------------------------------------------------------
2022-09-27T15:49:56.6299143Z Ran 1 test in 0.002s
2022-09-27T15:49:56.6299354Z 
2022-09-27T15:49:56.6299480Z OK (skipped=1)
2022-09-27T15:49:56.6299600Z 
2022-09-27T15:49:56.6299687Z Generating XML reports...
2022-09-27T15:49:56.6330142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154956.xml
2022-09-27T15:49:57.8189829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:57.8190219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:57.8197629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:57.8198130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:58.1605475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:58.1615115Z 
2022-09-27T15:49:58.1615241Z Running tests...
2022-09-27T15:49:58.1615814Z ----------------------------------------------------------------------
2022-09-27T15:49:58.1634430Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T15:49:58.1634827Z 
2022-09-27T15:49:58.1635234Z ----------------------------------------------------------------------
2022-09-27T15:49:58.1635640Z Ran 1 test in 0.002s
2022-09-27T15:49:58.1635836Z 
2022-09-27T15:49:58.1635959Z OK (skipped=1)
2022-09-27T15:49:58.1636149Z 
2022-09-27T15:49:58.1636301Z Generating XML reports...
2022-09-27T15:49:58.1666030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154958.xml
2022-09-27T15:49:59.3607989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:59.3608624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:59.3616765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:59.3617411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:59.6993649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:59.7003232Z 
2022-09-27T15:49:59.7003331Z Running tests...
2022-09-27T15:49:59.7004106Z ----------------------------------------------------------------------
2022-09-27T15:49:59.9336382Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18038
2022-09-27T15:49:59.9362890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18039
2022-09-27T15:49:59.9390004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18040
2022-09-27T15:50:00.9749160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.9749579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.9765358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.9765901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.9892790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.9893271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:01.0115562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:01.0116029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:01.0131102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:01.0131604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:01.0270628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:01.0271099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:01.3747967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:01.3849199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:01.3849857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:01.3850669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:01.3851376Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:01.3852310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:01.3856846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:01.3857408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:01.3858499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:01.3863282Z STAGE:2022-09-27 15:50:01 18038:18038 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:50:01 18040:18040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:01.3863613Z 
2022-09-27T15:50:01.3864453Z STAGE:2022-09-27 15:50:01 18039:18039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:01.3885793Z STAGE:2022-09-27 15:50:01 18040:18040 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:01.3888249Z STAGE:2022-09-27 15:50:01 18039:18039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:01.3889080Z STAGE:2022-09-27 15:50:01 18038:18038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:01.7441594Z ok (2.043s)
2022-09-27T15:50:01.7441851Z 
2022-09-27T15:50:01.7442280Z ----------------------------------------------------------------------
2022-09-27T15:50:01.7442545Z Ran 1 test in 2.044s
2022-09-27T15:50:01.7442664Z 
2022-09-27T15:50:01.7442729Z OK
2022-09-27T15:50:01.7442821Z 
2022-09-27T15:50:01.7442913Z Generating XML reports...
2022-09-27T15:50:01.7487153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154959.xml
2022-09-27T15:50:03.0769872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:03.0770229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:03.0778027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:03.0778490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:03.4183182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:03.4192863Z 
2022-09-27T15:50:03.4193028Z Running tests...
2022-09-27T15:50:03.4193667Z ----------------------------------------------------------------------
2022-09-27T15:50:03.6576191Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18094
2022-09-27T15:50:03.6602633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18095
2022-09-27T15:50:03.6629831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18096
2022-09-27T15:50:04.7777539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:04.7777927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:04.7934155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:04.7934820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:04.8126596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:04.8127036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:04.8242842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:04.8243178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:04.8268908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:04.8269501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:04.8560505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:04.8561137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:05.1791199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:05.1891556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:05.1892465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:05.1893302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:05.1894196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:05.1894995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:05.1997804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:05.2903445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:05.2904122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:05.5680348Z ok (2.148s)
2022-09-27T15:50:05.5680531Z 
2022-09-27T15:50:05.5680893Z ----------------------------------------------------------------------
2022-09-27T15:50:05.5681204Z Ran 1 test in 2.149s
2022-09-27T15:50:05.5681336Z 
2022-09-27T15:50:05.5681398Z OK
2022-09-27T15:50:05.5681491Z 
2022-09-27T15:50:05.5681583Z Generating XML reports...
2022-09-27T15:50:05.5726933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155003.xml
2022-09-27T15:50:06.8955347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:06.8955730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:06.8963588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:06.8963969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:07.2352642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:07.2362735Z 
2022-09-27T15:50:07.2363173Z Running tests...
2022-09-27T15:50:07.2363612Z ----------------------------------------------------------------------
2022-09-27T15:50:07.4731811Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18147
2022-09-27T15:50:07.4758272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18148
2022-09-27T15:50:07.4785096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18149
2022-09-27T15:50:08.5536530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:08.5536921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:08.5542447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:08.5542836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:08.5672285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:08.5672658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:08.5894125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:08.5894508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:08.5900283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:08.5900676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:08.6016758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:08.6017254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:08.9171440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:08.9186670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:08.9187131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:08.9187807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:08.9188376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:08.9271613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:08.9294179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:08.9294723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:08.9297883Z STAGE:2022-09-27 15:50:08 18149:18149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:08.9298320Z STAGE:2022-09-27 15:50:08 18148:18148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:09.0279944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:09.0283379Z STAGE:2022-09-27 15:50:09 18147:18147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:09.0298403Z STAGE:2022-09-27 15:50:09 18147:18147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:09.0301634Z STAGE:2022-09-27 15:50:09 18148:18148 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:09.0302484Z STAGE:2022-09-27 15:50:09 18149:18149 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:09.3835385Z ok (2.147s)
2022-09-27T15:50:09.3835612Z 
2022-09-27T15:50:09.3836059Z ----------------------------------------------------------------------
2022-09-27T15:50:09.3836471Z Ran 1 test in 2.147s
2022-09-27T15:50:09.3836653Z 
2022-09-27T15:50:09.3836747Z OK
2022-09-27T15:50:09.3836885Z 
2022-09-27T15:50:09.3837023Z Generating XML reports...
2022-09-27T15:50:09.3881061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155007.xml
2022-09-27T15:50:10.6911116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:10.6911542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:10.6919074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:10.6919683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:11.0331968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:11.0341238Z 
2022-09-27T15:50:11.0341382Z Running tests...
2022-09-27T15:50:11.0341830Z ----------------------------------------------------------------------
2022-09-27T15:50:11.2737806Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18203
2022-09-27T15:50:11.2763812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18204
2022-09-27T15:50:11.2791153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18205
2022-09-27T15:50:12.3693423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:12.3694069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:12.3858139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:12.3858513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:12.3889602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:12.3890281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:12.4036910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:12.4037496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:12.4216026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:12.4216668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:12.4244185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:12.4244818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:12.7569437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:12.7670929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:12.7671338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:12.7671958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:12.7672481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:12.7673024Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:12.7677723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:12.7678308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:12.7679311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:12.7682797Z STAGE:2022-09-27 15:50:12 18203:18203 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:12.7683808Z STAGE:2022-09-27 15:50:12 18204:18204 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:12.7684879Z STAGE:2022-09-27 15:50:12 18205:18205 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:12.7696569Z STAGE:2022-09-27 15:50:12 18203:18203 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:12.7698859Z STAGE:2022-09-27 15:50:12 18205:18205 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:12.7699868Z STAGE:2022-09-27 15:50:12 18204:18204 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:13.0838630Z ok (2.049s)
2022-09-27T15:50:13.0839054Z 
2022-09-27T15:50:13.0839520Z ----------------------------------------------------------------------
2022-09-27T15:50:13.0839908Z Ran 1 test in 2.050s
2022-09-27T15:50:13.0840091Z 
2022-09-27T15:50:13.0840187Z OK
2022-09-27T15:50:13.0840311Z 
2022-09-27T15:50:13.0840455Z Generating XML reports...
2022-09-27T15:50:13.0885650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155011.xml
2022-09-27T15:50:14.3880567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:14.3881018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:14.3888418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:14.3888936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:14.7267980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:14.7277745Z 
2022-09-27T15:50:14.7277844Z Running tests...
2022-09-27T15:50:14.7278397Z ----------------------------------------------------------------------
2022-09-27T15:50:14.9629887Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18259
2022-09-27T15:50:14.9656322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18260
2022-09-27T15:50:14.9683099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18261
2022-09-27T15:50:16.0930498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:16.0931154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:16.0931954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:16.0932576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:16.1274400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:16.1275114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:16.1281146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:16.1281714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:16.1417926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:16.1418527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:16.1734962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:16.1735623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:16.5166283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:16.5166726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:16.5167224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:16.5168203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:16.5169047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:16.5170096Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:16.5272410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:16.6177789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:16.6178186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:16.9734667Z ok (2.245s)
2022-09-27T15:50:16.9736371Z 
2022-09-27T15:50:16.9736812Z ----------------------------------------------------------------------
2022-09-27T15:50:16.9737195Z Ran 1 test in 2.246s
2022-09-27T15:50:16.9739124Z 
2022-09-27T15:50:16.9739332Z OK
2022-09-27T15:50:16.9739477Z 
2022-09-27T15:50:16.9739611Z Generating XML reports...
2022-09-27T15:50:16.9787429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155014.xml
2022-09-27T15:50:18.2769223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:18.2774591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:18.2778763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:18.2779385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:18.6171180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:18.6181314Z 
2022-09-27T15:50:18.6181645Z Running tests...
2022-09-27T15:50:18.6182231Z ----------------------------------------------------------------------
2022-09-27T15:50:18.8547796Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18339
2022-09-27T15:50:18.8574402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18340
2022-09-27T15:50:18.8601907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18341
2022-09-27T15:50:19.9037211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:19.9037626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:19.9199787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:19.9200300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:19.9378387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:19.9378762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:19.9541352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:19.9542054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:19.9561814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:19.9562463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:19.9877906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:19.9878314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:20.3171933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:20.3172696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:20.3173376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:20.3174402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:20.3174928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:20.3175527Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:20.3180809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:20.3181402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:20.3181975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:20.6651110Z skip: CUDA is not available. (2.047s)
2022-09-27T15:50:20.6651413Z 
2022-09-27T15:50:20.6651926Z ----------------------------------------------------------------------
2022-09-27T15:50:20.6652245Z Ran 1 test in 2.047s
2022-09-27T15:50:20.6652361Z 
2022-09-27T15:50:20.6652439Z OK (skipped=1)
2022-09-27T15:50:20.6652566Z 
2022-09-27T15:50:20.6652640Z Generating XML reports...
2022-09-27T15:50:20.6697702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155018.xml
2022-09-27T15:50:21.9817359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:21.9817991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:21.9826116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:21.9826746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:22.3205589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:22.3215501Z 
2022-09-27T15:50:22.3216066Z Running tests...
2022-09-27T15:50:22.3216533Z ----------------------------------------------------------------------
2022-09-27T15:50:22.5578557Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18392
2022-09-27T15:50:22.5604872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18393
2022-09-27T15:50:22.5631603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18394
2022-09-27T15:50:23.6547559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.6548187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.6878732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.6879380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.6933945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.6934606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.7023364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.7024095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.7292869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.7293481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.7371432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.7372107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:24.0865413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:24.0866364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:24.0867046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:24.0867888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:24.0868422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:24.0868946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:24.0873822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:24.0875148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:24.0875888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:24.3677489Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:50:24.3677761Z 
2022-09-27T15:50:24.3678201Z ----------------------------------------------------------------------
2022-09-27T15:50:24.3678688Z Ran 1 test in 2.046s
2022-09-27T15:50:24.3678886Z 
2022-09-27T15:50:24.3678951Z OK (skipped=1)
2022-09-27T15:50:24.3679061Z 
2022-09-27T15:50:24.3679147Z Generating XML reports...
2022-09-27T15:50:24.3723413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155022.xml
2022-09-27T15:50:25.6577158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:25.6577795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:25.6585315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:25.6585696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:26.0097598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:26.0107056Z 
2022-09-27T15:50:26.0107159Z Running tests...
2022-09-27T15:50:26.0107771Z ----------------------------------------------------------------------
2022-09-27T15:50:26.2461169Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18445
2022-09-27T15:50:26.2486780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18446
2022-09-27T15:50:26.2513446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18447
2022-09-27T15:50:27.3331836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.3332479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.3353279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.3353906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.3369334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.3369736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.3687589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.3687980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.3716553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.3716952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.3724210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.3724586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.7090391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:27.7191648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:27.7192139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:27.7192832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:27.7193698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:27.7194451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:27.7199284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:27.7199800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:27.7200970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:27.7259392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_ctp3s2
2022-09-27T15:50:27.7261450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_ctp3s2/_remote_module_non_scriptable.py
2022-09-27T15:50:27.7262131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdfpcfv7w
2022-09-27T15:50:27.7262773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucogzfrj
2022-09-27T15:50:27.7263605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucogzfrj/_remote_module_non_scriptable.py
2022-09-27T15:50:27.7264409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdfpcfv7w/_remote_module_non_scriptable.py
2022-09-27T15:50:28.0560764Z ok (2.045s)
2022-09-27T15:50:28.0560987Z 
2022-09-27T15:50:28.0561420Z ----------------------------------------------------------------------
2022-09-27T15:50:28.0561716Z Ran 1 test in 2.045s
2022-09-27T15:50:28.0561860Z 
2022-09-27T15:50:28.0561953Z OK
2022-09-27T15:50:28.0562059Z 
2022-09-27T15:50:28.0562155Z Generating XML reports...
2022-09-27T15:50:28.0606968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155026.xml
2022-09-27T15:50:29.3439786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:29.3440163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:29.3448510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:29.3448871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:29.6822741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:29.6832937Z 
2022-09-27T15:50:29.6833387Z Running tests...
2022-09-27T15:50:29.6833785Z ----------------------------------------------------------------------
2022-09-27T15:50:29.9176800Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18498
2022-09-27T15:50:29.9204344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18499
2022-09-27T15:50:29.9231460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18500
2022-09-27T15:50:30.9649425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:30.9650085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:30.9811096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:30.9811817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:30.9869660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:30.9870426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:30.9994703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:30.9995218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.0241927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:31.0242329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.0270333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:31.0271075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.3638955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:31.3639474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:31.3639851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:31.3640477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:31.3641018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:31.3641552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:31.3647344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:31.4651476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:31.4651928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:31.7281479Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:50:31.7281815Z 
2022-09-27T15:50:31.7282211Z ----------------------------------------------------------------------
2022-09-27T15:50:31.7282467Z Ran 1 test in 2.045s
2022-09-27T15:50:31.7282582Z 
2022-09-27T15:50:31.7282643Z OK (skipped=1)
2022-09-27T15:50:31.7282753Z 
2022-09-27T15:50:31.7282843Z Generating XML reports...
2022-09-27T15:50:31.7329210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155029.xml
2022-09-27T15:50:33.0198738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:33.0199143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:33.0207102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:33.0207484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:33.3666735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:33.3676740Z 
2022-09-27T15:50:33.3677046Z Running tests...
2022-09-27T15:50:33.3677715Z ----------------------------------------------------------------------
2022-09-27T15:50:33.6024021Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18551
2022-09-27T15:50:33.6050039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18552
2022-09-27T15:50:33.6077363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18553
2022-09-27T15:50:34.6845132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.6845799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.6972281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.6972883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.6980411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.6981049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.7192677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.7193253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:34.7326993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.7327737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:34.7338818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.7339232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:35.0586223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:35.0687529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:35.0688233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:35.0688907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:35.0689459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:35.0690163Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:35.0695869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:35.0696329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:35.0698733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:35.4127733Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:50:35.4128013Z 
2022-09-27T15:50:35.4128466Z ----------------------------------------------------------------------
2022-09-27T15:50:35.4128836Z Ran 1 test in 2.045s
2022-09-27T15:50:35.4129019Z 
2022-09-27T15:50:35.4129136Z OK (skipped=1)
2022-09-27T15:50:35.4129303Z 
2022-09-27T15:50:35.4129459Z Generating XML reports...
2022-09-27T15:50:35.4175258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155033.xml
2022-09-27T15:50:36.6971838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:36.6972230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:36.6979941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:36.6980308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:37.0341586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:37.0351928Z 
2022-09-27T15:50:37.0352078Z Running tests...
2022-09-27T15:50:37.0352695Z ----------------------------------------------------------------------
2022-09-27T15:50:37.0369702Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:50:37.0369990Z 
2022-09-27T15:50:37.0370208Z ----------------------------------------------------------------------
2022-09-27T15:50:37.0370534Z Ran 1 test in 0.002s
2022-09-27T15:50:37.0370649Z 
2022-09-27T15:50:37.0370710Z OK (skipped=1)
2022-09-27T15:50:37.0370820Z 
2022-09-27T15:50:37.0370907Z Generating XML reports...
2022-09-27T15:50:37.0400212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155037.xml
2022-09-27T15:50:38.2222661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:38.2223299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:38.2231878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:38.2232529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:38.5656938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:38.5667328Z 
2022-09-27T15:50:38.5667687Z Running tests...
2022-09-27T15:50:38.5668355Z ----------------------------------------------------------------------
2022-09-27T15:50:38.5683713Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:50:38.5684123Z 
2022-09-27T15:50:38.5684345Z ----------------------------------------------------------------------
2022-09-27T15:50:38.5684591Z Ran 1 test in 0.002s
2022-09-27T15:50:38.5684692Z 
2022-09-27T15:50:38.5684767Z OK (skipped=1)
2022-09-27T15:50:38.5684877Z 
2022-09-27T15:50:38.5684962Z Generating XML reports...
2022-09-27T15:50:38.5714348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155038.xml
2022-09-27T15:50:39.0600082Z Running distributed tests for the test backend with file init_method
2022-09-27T15:50:39.0613540Z Running distributed tests for the gloo backend with file init_method
2022-09-27T15:50:39.0619704Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:50:39.061625]
2022-09-27T15:50:39.0627818Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:50:39.062476]
2022-09-27T15:50:40.4343106Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:50:40.4382398Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T15:50:40.4417918Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4418504Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4419027Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4419522Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4420039Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4420576Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4421189Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4421780Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4422501Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4423208Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4424053Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4424762Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4425460Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4426127Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4426778Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4427413Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4427993Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4428528Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4429038Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4429572Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4430168Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4430744Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4431256Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4431777Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4432525Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4433059Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4433572Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4434213Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4434731Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4435212Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4435732Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4436282Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4436785Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4437307Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4437848Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4438418Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4438975Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4439518Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4440077Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4440656Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4441269Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4441883Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4442457Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4443001Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4443542Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4444112Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4444646Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4445239Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4445841Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4446386Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4446931Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4447489Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4448058Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4448573Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4449118Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4449650Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4450160Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4450675Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4451200Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4451721Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4452203Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4452707Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4453198Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4453705Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4454221Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4454735Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4455194Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4455817Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4456331Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4456934Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4457440Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4457998Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4458495Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4458956Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4459440Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4459956Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4460480Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4461015Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4461512Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4462012Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4462533Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4463129Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4463697Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4464418Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4465083Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4465681Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4466251Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4466827Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4467410Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4467979Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4468518Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4469123Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4469693Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4470249Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4470780Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4471343Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4471914Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4472420Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4472908Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4473370Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4473826Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4474300Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4474808Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4475317Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4475789Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4476311Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4476839Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4477339Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4477859Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4478523Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4479065Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4479713Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4480274Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4480804Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4481304Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4481833Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4482408Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4482925Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4483430Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4483913Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4484428Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4484943Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4485433Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4485925Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4486511Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4487192Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4487800Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4488307Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4488939Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4489494Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4490079Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4490675Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4491273Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4491826Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4492408Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4492944Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4493419Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4493906Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4494446Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4494986Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4495553Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4496119Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4496647Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4497185Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4497774Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4498443Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4499156Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4499974Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4500846Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4501693Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4502615Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4503431Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4504380Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4505260Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4505978Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4506637Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4507211Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4507715Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4508247Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4508769Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4509263Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4509809Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4510378Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4510990Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4511624Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4512182Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4512648Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4513192Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4513785Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4514368Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4514903Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4515454Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4516036Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4516608Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4517178Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4517727Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4518281Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4518840Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4519391Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4519945Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4520523Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4521054Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4521537Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4522011Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4522567Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4523297Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4523782Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4524297Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4524765Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4525243Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4525717Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4526187Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4526718Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4527213Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4527665Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4528131Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4528625Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4529114Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4529612Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4530091Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4530524Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4531000Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4531508Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4532003Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4532590Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4533132Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4533629Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4534158Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4534762Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4535306Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4535840Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4536375Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4536911Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4537419Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4537926Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4538440Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4538967Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4539562Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4540170Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4540777Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4541336Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4541945Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4542538Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4543075Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4543570Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4544279Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4544838Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4545586Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4546233Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4546986Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4547648Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4548167Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4548637Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4549133Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4549642Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4550152Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4550632Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4551135Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4551607Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4552080Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4552552Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4553029Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4553517Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4554010Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4554472Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4554953Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4555453Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4555933Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4556402Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4556888Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4557397Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4557875Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4558386Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4558890Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4559344Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4559832Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4560316Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4560805Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4561349Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4561942Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4562523Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4562999Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4563521Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4564096Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4564609Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4565115Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4565669Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4566235Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4566788Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4567318Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4567860Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4568483Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4568986Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4569520Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4570159Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4570752Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:40.4716288Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:50:41.4510505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:41.4510888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:41.4518300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:41.4518809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:41.7905866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:41.7914998Z 
2022-09-27T15:50:41.7915418Z Running tests...
2022-09-27T15:50:41.7915839Z ----------------------------------------------------------------------
2022-09-27T15:50:42.0253905Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18649
2022-09-27T15:50:42.0280058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18650
2022-09-27T15:50:42.0306263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18651
2022-09-27T15:50:43.1209326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:43.1210076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:43.1339148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:43.1339682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:43.1358923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:43.1359499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:43.1556861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:43.1557271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:43.1694475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:43.1694958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:43.1715599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:43.1716130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:43.5066961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:43.5167766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:43.5168267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:43.5169049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:43.5169586Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:43.5170113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:43.5176400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:43.5177053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:43.5177698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:43.8354568Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:50:43.8354884Z 
2022-09-27T15:50:43.8355407Z ----------------------------------------------------------------------
2022-09-27T15:50:43.8355770Z Ran 1 test in 2.044s
2022-09-27T15:50:43.8355873Z 
2022-09-27T15:50:43.8355948Z OK (skipped=1)
2022-09-27T15:50:43.8356060Z 
2022-09-27T15:50:43.8356146Z Generating XML reports...
2022-09-27T15:50:43.8402801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155041.xml
2022-09-27T15:50:45.1196745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:45.1197116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:45.1204687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:45.1205117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:45.4687688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:45.4697053Z 
2022-09-27T15:50:45.4697192Z Running tests...
2022-09-27T15:50:45.4697623Z ----------------------------------------------------------------------
2022-09-27T15:50:45.4733906Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-09-27T15:50:45.4734352Z 
2022-09-27T15:50:45.4734649Z ----------------------------------------------------------------------
2022-09-27T15:50:45.4734900Z Ran 1 test in 0.004s
2022-09-27T15:50:45.4735016Z 
2022-09-27T15:50:45.4735100Z OK (skipped=1)
2022-09-27T15:50:45.4735215Z 
2022-09-27T15:50:45.4735287Z Generating XML reports...
2022-09-27T15:50:45.4765269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155045.xml
2022-09-27T15:50:46.6614672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:46.6615054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:46.6622663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:46.6623031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:46.9992970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:47.0002386Z 
2022-09-27T15:50:47.0002468Z Running tests...
2022-09-27T15:50:47.0002973Z ----------------------------------------------------------------------
2022-09-27T15:50:47.2352819Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18712
2022-09-27T15:50:47.2378100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18713
2022-09-27T15:50:47.2404976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18714
2022-09-27T15:50:48.2916440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.2916807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.3229128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.3229654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.3307645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.3307986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.3528318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.3529270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.3633621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.3634212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.3860776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.3861409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.7323893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:48.7324561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:48.7324987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:48.7326033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:48.7326655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:48.7327185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:48.7331757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:48.7332440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:48.8333592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:49.1454283Z ok (2.145s)
2022-09-27T15:50:49.1454521Z 
2022-09-27T15:50:49.1455031Z ----------------------------------------------------------------------
2022-09-27T15:50:49.1455490Z Ran 1 test in 2.145s
2022-09-27T15:50:49.1455679Z 
2022-09-27T15:50:49.1455796Z OK
2022-09-27T15:50:49.1455965Z 
2022-09-27T15:50:49.1456129Z Generating XML reports...
2022-09-27T15:50:49.1500942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155046.xml
2022-09-27T15:50:50.4505782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:50.4506157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:50.4513845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:50.4514283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:50.7926320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:50.7935791Z 
2022-09-27T15:50:50.7935935Z Running tests...
2022-09-27T15:50:50.7936541Z ----------------------------------------------------------------------
2022-09-27T15:50:51.0336477Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18765
2022-09-27T15:50:51.0362748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18766
2022-09-27T15:50:51.0388615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18767
2022-09-27T15:50:52.1300643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.1301343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.1629809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.1630413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.1701905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.1702529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.1779932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.2082455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.2083378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.2084024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.2155620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.2156258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.5630184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:52.5630791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:52.5631760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:52.5632339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:52.5633180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:52.5633837Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:52.5735806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:52.6645151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:52.6645727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:52.9439272Z skip: CUDA is not available. (2.150s)
2022-09-27T15:50:52.9439639Z 
2022-09-27T15:50:52.9440098Z ----------------------------------------------------------------------
2022-09-27T15:50:52.9440347Z Ran 1 test in 2.150s
2022-09-27T15:50:52.9440462Z 
2022-09-27T15:50:52.9440534Z OK (skipped=1)
2022-09-27T15:50:52.9440630Z 
2022-09-27T15:50:52.9440715Z Generating XML reports...
2022-09-27T15:50:52.9484757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155050.xml
2022-09-27T15:50:54.2702023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:54.2702480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:54.2709330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:54.2709700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:54.6091729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:54.6101689Z 
2022-09-27T15:50:54.6101797Z Running tests...
2022-09-27T15:50:54.6102243Z ----------------------------------------------------------------------
2022-09-27T15:50:54.8478498Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18818
2022-09-27T15:50:54.8505005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18819
2022-09-27T15:50:54.8532400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18820
2022-09-27T15:50:55.9557368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:55.9558317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:55.9715028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:55.9715760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:55.9757089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:55.9757754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:55.9899858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:55.9900393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:56.0072566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:56.0073177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:56.0105470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:56.0105880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:56.3482487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:56.3583207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:50:56.3583809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:56.3584971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:56.3585788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:56.3586707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:50:56.3690116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:50:56.3757028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dag2cxe
2022-09-27T15:50:56.3758913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dag2cxe/_remote_module_non_scriptable.py
2022-09-27T15:50:56.4594911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:56.4595422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:56.4665423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdd344t9w
2022-09-27T15:50:56.4666163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1dn6y24
2022-09-27T15:50:56.4667374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdd344t9w/_remote_module_non_scriptable.py
2022-09-27T15:50:56.4668438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1dn6y24/_remote_module_non_scriptable.py
2022-09-27T15:50:56.4844569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:56.4845268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:56.4845935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:56.8581540Z ok (2.248s)
2022-09-27T15:50:56.8581765Z 
2022-09-27T15:50:56.8582271Z ----------------------------------------------------------------------
2022-09-27T15:50:56.8582721Z Ran 1 test in 2.248s
2022-09-27T15:50:56.8582923Z 
2022-09-27T15:50:56.8583028Z OK
2022-09-27T15:50:56.8583192Z 
2022-09-27T15:50:56.8583622Z Generating XML reports...
2022-09-27T15:50:56.8628152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155054.xml
2022-09-27T15:50:58.1604078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:58.1604460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:58.1611792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:58.1612163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:58.5013779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:58.5022630Z 
2022-09-27T15:50:58.5022729Z Running tests...
2022-09-27T15:50:58.5023175Z ----------------------------------------------------------------------
2022-09-27T15:50:58.7378629Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18871
2022-09-27T15:50:58.7403624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18872
2022-09-27T15:50:58.7430047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18873
2022-09-27T15:50:59.8251427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:59.8252058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:59.8426753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:59.8427441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:59.8592110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:59.8592689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:59.8710770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:59.8711415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:59.8772687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:59.8773336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:59.9061500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:59.9061874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:00.2254369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:00.2354951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:00.2355480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:00.2356256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:00.2357115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:00.2357941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:00.2461188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:00.2525435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcul1g5i
2022-09-27T15:51:00.2527292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcul1g5i/_remote_module_non_scriptable.py
2022-09-27T15:51:00.3365342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:00.3366178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:00.3468525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1haryqi
2022-09-27T15:51:00.3468952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwgbmjhs
2022-09-27T15:51:00.3471260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1haryqi/_remote_module_non_scriptable.py
2022-09-27T15:51:00.3471749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwgbmjhs/_remote_module_non_scriptable.py
2022-09-27T15:51:00.3700092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:00.3700565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:00.3700951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:00.7479334Z ok (2.245s)
2022-09-27T15:51:00.7479524Z 
2022-09-27T15:51:00.7479892Z ----------------------------------------------------------------------
2022-09-27T15:51:00.7480249Z Ran 1 test in 2.246s
2022-09-27T15:51:00.7480368Z 
2022-09-27T15:51:00.7480429Z OK
2022-09-27T15:51:00.7480521Z 
2022-09-27T15:51:00.7480612Z Generating XML reports...
2022-09-27T15:51:00.7528220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155058.xml
2022-09-27T15:51:02.0621527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:02.0621881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:02.0629493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:02.0629863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:02.4004010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:02.4012498Z 
2022-09-27T15:51:02.4012638Z Running tests...
2022-09-27T15:51:02.4013043Z ----------------------------------------------------------------------
2022-09-27T15:51:02.6373777Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18924
2022-09-27T15:51:02.6400480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18925
2022-09-27T15:51:02.6427442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18926
2022-09-27T15:51:03.6689768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:03.6690455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:03.6727807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:03.6728430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:03.7033102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:03.7033786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:03.7072303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:03.7072911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:03.7142566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:03.7143212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:03.7465256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:03.7465676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:04.0978557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:04.0979206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:04.0979851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:04.0980482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:04.0981349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:04.0982201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:04.0986964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:04.0987630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:04.0988024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:04.4474421Z skip: CUDA is not available. (2.046s)
2022-09-27T15:51:04.4474749Z 
2022-09-27T15:51:04.4475249Z ----------------------------------------------------------------------
2022-09-27T15:51:04.4475498Z Ran 1 test in 2.046s
2022-09-27T15:51:04.4475615Z 
2022-09-27T15:51:04.4475677Z OK (skipped=1)
2022-09-27T15:51:04.4475786Z 
2022-09-27T15:51:04.4475872Z Generating XML reports...
2022-09-27T15:51:04.4522516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155102.xml
2022-09-27T15:51:05.7454187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:05.7454565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:05.7462013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:05.7462369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:06.0856950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:06.0867403Z 
2022-09-27T15:51:06.0867832Z Running tests...
2022-09-27T15:51:06.0868431Z ----------------------------------------------------------------------
2022-09-27T15:51:06.3226668Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18977
2022-09-27T15:51:06.3252992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18978
2022-09-27T15:51:06.3279979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18979
2022-09-27T15:51:07.3805553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:07.3806198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:07.3853682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:07.3854331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:07.4086980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:07.4087347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:07.4146094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:07.4146625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:07.4206583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:07.4207041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:07.4416358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:07.4416784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:07.7752916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:07.7822416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:07.7822845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:07.7823471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:07.7824151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:07.7853976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:07.7930034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:07.7930421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:07.8866177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:08.1330573Z skip: CUDA is not available. (2.046s)
2022-09-27T15:51:08.1330770Z 
2022-09-27T15:51:08.1331082Z ----------------------------------------------------------------------
2022-09-27T15:51:08.1331361Z Ran 1 test in 2.046s
2022-09-27T15:51:08.1331478Z 
2022-09-27T15:51:08.1331553Z OK (skipped=1)
2022-09-27T15:51:08.1331675Z 
2022-09-27T15:51:08.1331762Z Generating XML reports...
2022-09-27T15:51:08.1377167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155106.xml
2022-09-27T15:51:09.4328455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:09.4329138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:09.4337340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:09.7727077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:09.7727956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:09.7737398Z 
2022-09-27T15:51:09.7737797Z Running tests...
2022-09-27T15:51:09.7738272Z ----------------------------------------------------------------------
2022-09-27T15:51:10.0075987Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19030
2022-09-27T15:51:10.0101817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19031
2022-09-27T15:51:10.0128553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19032
2022-09-27T15:51:11.1214823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:11.1215214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:11.1325777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:11.1326300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:11.1560574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:11.1560945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:11.1604691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:11.1605576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:11.1675496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:11.1676046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:11.1940887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:11.1941303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:11.5273956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:11.5374869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:11.5375458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:11.5376073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:11.5376612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:11.5377138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:11.5382644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:11.5383744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:11.5384383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:11.8178631Z skip: CUDA is not available. (2.044s)
2022-09-27T15:51:11.8178894Z 
2022-09-27T15:51:11.8179212Z ----------------------------------------------------------------------
2022-09-27T15:51:11.8179475Z Ran 1 test in 2.044s
2022-09-27T15:51:11.8179591Z 
2022-09-27T15:51:11.8179669Z OK (skipped=1)
2022-09-27T15:51:11.8179839Z 
2022-09-27T15:51:11.8179927Z Generating XML reports...
2022-09-27T15:51:11.8225593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155109.xml
2022-09-27T15:51:13.1104487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:13.1104937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:13.1113943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:13.1114365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:13.4531609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:13.4541605Z 
2022-09-27T15:51:13.4541914Z Running tests...
2022-09-27T15:51:13.4542503Z ----------------------------------------------------------------------
2022-09-27T15:51:13.6899949Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19083
2022-09-27T15:51:13.6926704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19084
2022-09-27T15:51:13.6952916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19085
2022-09-27T15:51:14.7813113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.7813506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.7939683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.7940028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.8151657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.8152137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.8268911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.8269500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.8655478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.8656029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.8978075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.8978432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:15.2325143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:15.2348285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:15.2348731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:15.2349341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:15.2349872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:15.2426840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:15.2455442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:15.2456493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:15.3438709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:15.6004572Z skip: CUDA is not available. (2.146s)
2022-09-27T15:51:15.6004820Z 
2022-09-27T15:51:15.6005107Z ----------------------------------------------------------------------
2022-09-27T15:51:15.6005364Z Ran 1 test in 2.146s
2022-09-27T15:51:15.6005480Z 
2022-09-27T15:51:15.6005554Z OK (skipped=1)
2022-09-27T15:51:15.6005663Z 
2022-09-27T15:51:15.6005750Z Generating XML reports...
2022-09-27T15:51:15.6051131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155113.xml
2022-09-27T15:51:16.8941468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:16.8941851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:16.8949490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:16.8949910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.2326099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:17.2335519Z 
2022-09-27T15:51:17.2335611Z Running tests...
2022-09-27T15:51:17.2336412Z ----------------------------------------------------------------------
2022-09-27T15:51:17.2355939Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T15:51:17.2356425Z 
2022-09-27T15:51:17.2356837Z ----------------------------------------------------------------------
2022-09-27T15:51:17.2357550Z Ran 1 test in 0.002s
2022-09-27T15:51:17.2357683Z 
2022-09-27T15:51:17.2357747Z OK (skipped=1)
2022-09-27T15:51:17.2357862Z 
2022-09-27T15:51:17.2358412Z Generating XML reports...
2022-09-27T15:51:17.2387071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml
2022-09-27T15:51:18.4294285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:18.4294664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:18.4306686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:18.4307049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:18.7951298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:18.7961789Z 
2022-09-27T15:51:18.7962153Z Running tests...
2022-09-27T15:51:18.7962768Z ----------------------------------------------------------------------
2022-09-27T15:51:19.0353984Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19146
2022-09-27T15:51:19.0380077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19147
2022-09-27T15:51:19.0407741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19148
2022-09-27T15:51:20.0810490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:20.0811129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:20.1094302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:20.1094940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:20.1130958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:20.1131599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:20.1262106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:20.1262741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:20.1436847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:20.1437387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:20.1596384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:20.1597001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:20.4918636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:20.5019343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:20.5020015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:20.5021000Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:20.5021903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:20.5022719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:20.5125569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:20.6033207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:20.6033640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:20.8453790Z skip: CUDA is not available. (2.049s)
2022-09-27T15:51:20.8454366Z 
2022-09-27T15:51:20.8454866Z ----------------------------------------------------------------------
2022-09-27T15:51:20.8455144Z Ran 1 test in 2.049s
2022-09-27T15:51:20.8455264Z 
2022-09-27T15:51:20.8455337Z OK (skipped=1)
2022-09-27T15:51:20.8455445Z 
2022-09-27T15:51:20.8455517Z Generating XML reports...
2022-09-27T15:51:20.8499726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155118.xml
2022-09-27T15:51:22.1408206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:22.1408587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:22.1415720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:22.1416085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:22.4791267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:22.4801215Z 
2022-09-27T15:51:22.4801548Z Running tests...
2022-09-27T15:51:22.4801941Z ----------------------------------------------------------------------
2022-09-27T15:51:22.7241658Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19199
2022-09-27T15:51:22.7267739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19200
2022-09-27T15:51:22.7294018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19201
2022-09-27T15:51:23.8015413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:23.8015818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:23.8018358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:23.8018879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:23.8272118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:23.8272607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:23.8364830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:23.8365227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:23.8374824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:23.8375233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:23.8608046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:23.8608435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:24.1837000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:24.1936836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:24.1937532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:24.1938379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:24.1938912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:24.1939615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:24.1946047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:24.1946800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:24.1947354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:24.5342952Z skip: CUDA is not available. (2.054s)
2022-09-27T15:51:24.5343222Z 
2022-09-27T15:51:24.5343545Z ----------------------------------------------------------------------
2022-09-27T15:51:24.5343802Z Ran 1 test in 2.054s
2022-09-27T15:51:24.5344058Z 
2022-09-27T15:51:24.5344134Z OK (skipped=1)
2022-09-27T15:51:24.5344245Z 
2022-09-27T15:51:24.5344333Z Generating XML reports...
2022-09-27T15:51:24.5389839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155122.xml
2022-09-27T15:51:25.8139117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:25.8139517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:25.8147936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:25.8148297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:26.1530732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:26.1540470Z 
2022-09-27T15:51:26.1540793Z Running tests...
2022-09-27T15:51:26.1541396Z ----------------------------------------------------------------------
2022-09-27T15:51:26.3801318Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.226s)
2022-09-27T15:51:26.3802305Z 
2022-09-27T15:51:26.3802679Z ----------------------------------------------------------------------
2022-09-27T15:51:26.3803104Z Ran 1 test in 0.226s
2022-09-27T15:51:26.3803281Z 
2022-09-27T15:51:26.3803404Z OK (skipped=1)
2022-09-27T15:51:26.3803584Z 
2022-09-27T15:51:26.3803728Z Generating XML reports...
2022-09-27T15:51:26.3825867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155126.xml
2022-09-27T15:51:27.6214429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:27.6215067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:27.6223685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:27.6224627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:27.9600823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:27.9610631Z 
2022-09-27T15:51:27.9610935Z Running tests...
2022-09-27T15:51:27.9611565Z ----------------------------------------------------------------------
2022-09-27T15:51:28.1955994Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19262
2022-09-27T15:51:28.1983286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19263
2022-09-27T15:51:28.2010270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19264
2022-09-27T15:51:29.2852803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:29.2853244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:29.3056205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:29.3056769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:29.3195413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:29.3195987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:29.3337329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:29.3337757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:29.3393121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:29.3393751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:29.3655566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:29.3655969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:29.6907986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:29.6908677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:29.6909036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:29.6909656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:29.6910185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:29.6910726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:29.7014335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:29.7918980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:29.7920068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:30.2063755Z ok (2.245s)
2022-09-27T15:51:30.2064111Z 
2022-09-27T15:51:30.2064541Z ----------------------------------------------------------------------
2022-09-27T15:51:30.2064916Z Ran 1 test in 2.245s
2022-09-27T15:51:30.2065153Z 
2022-09-27T15:51:30.2065259Z OK
2022-09-27T15:51:30.2065408Z 
2022-09-27T15:51:30.2066195Z Generating XML reports...
2022-09-27T15:51:30.2110276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155127.xml
2022-09-27T15:51:31.5360150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:31.5360778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:31.5368529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:31.5369157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:31.8772294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:31.8782366Z 
2022-09-27T15:51:31.8782825Z Running tests...
2022-09-27T15:51:31.8783221Z ----------------------------------------------------------------------
2022-09-27T15:51:32.1141120Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19315
2022-09-27T15:51:32.1168343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19316
2022-09-27T15:51:32.1195010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19317
2022-09-27T15:51:33.2141441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:33.2141838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:33.2162562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:33.2162913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:33.2487303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:33.2487929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:33.2515369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:33.2515864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:33.2563206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:33.2563737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:33.2888434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:33.2888825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:33.6058255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:33.6159114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:33.6159755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:33.6160402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:33.6160943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:33.6161481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:33.6265858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:33.7172540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:33.7172978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:34.0243280Z skip: CUDA is not available. (2.146s)
2022-09-27T15:51:34.0243499Z 
2022-09-27T15:51:34.0243819Z ----------------------------------------------------------------------
2022-09-27T15:51:34.0244087Z Ran 1 test in 2.146s
2022-09-27T15:51:34.0244204Z 
2022-09-27T15:51:34.0244280Z OK (skipped=1)
2022-09-27T15:51:34.0244390Z 
2022-09-27T15:51:34.0244467Z Generating XML reports...
2022-09-27T15:51:34.0290095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155131.xml
2022-09-27T15:51:35.3299074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:35.3299525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:35.3307203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:35.3307580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:35.6683455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:35.6692948Z 
2022-09-27T15:51:35.6693064Z Running tests...
2022-09-27T15:51:35.6693750Z ----------------------------------------------------------------------
2022-09-27T15:51:35.9055274Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19368
2022-09-27T15:51:35.9081412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19369
2022-09-27T15:51:35.9109130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19370
2022-09-27T15:51:37.0148577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:37.0149244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:37.0363299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:37.0363825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:37.0512184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:37.0512567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:37.0523595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:37.0524248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:37.0714783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:37.0715342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:37.0863716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:37.0864552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:37.4279419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:37.4380695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:37.4381118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:37.4381761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:37.4382405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:37.4383286Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:37.4389037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:37.4390076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:37.4391025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:37.7156269Z skip: Need at least 3 CUDA devices (2.046s)
2022-09-27T15:51:37.7156501Z 
2022-09-27T15:51:37.7156945Z ----------------------------------------------------------------------
2022-09-27T15:51:37.7157237Z Ran 1 test in 2.046s
2022-09-27T15:51:37.7157356Z 
2022-09-27T15:51:37.7157431Z OK (skipped=1)
2022-09-27T15:51:37.7157526Z 
2022-09-27T15:51:37.7157622Z Generating XML reports...
2022-09-27T15:51:37.7202129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155135.xml
2022-09-27T15:51:39.0213517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:39.0213898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:39.0221741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:39.0222160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:39.3683626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:39.3692645Z 
2022-09-27T15:51:39.3692780Z Running tests...
2022-09-27T15:51:39.3693370Z ----------------------------------------------------------------------
2022-09-27T15:51:39.3710414Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T15:51:39.3710792Z 
2022-09-27T15:51:39.3711339Z ----------------------------------------------------------------------
2022-09-27T15:51:39.3711836Z Ran 1 test in 0.002s
2022-09-27T15:51:39.3711998Z 
2022-09-27T15:51:39.3712082Z OK (skipped=1)
2022-09-27T15:51:39.3712180Z 
2022-09-27T15:51:39.3712269Z Generating XML reports...
2022-09-27T15:51:39.3740767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml
2022-09-27T15:51:40.5847855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:40.5848259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:40.5855456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:40.5855893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:40.9247205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:40.9257175Z 
2022-09-27T15:51:40.9257277Z Running tests...
2022-09-27T15:51:40.9257817Z ----------------------------------------------------------------------
2022-09-27T15:51:40.9273765Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:51:41.1674943Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19431
2022-09-27T15:51:41.1700088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19432
2022-09-27T15:51:41.1728410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19433
2022-09-27T15:51:42.2494620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:42.2495268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:42.2819119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:42.2819515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:42.2866524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:42.2867047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:42.3092055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:42.3092389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:42.3194218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:42.3194589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:42.3419009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:42.3419683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:42.6578237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:42.6679649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:42.6680776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:42.6681207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:42.6681794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:42.6682522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:42.6688712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:42.6689331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:42.6757579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3xeicls
2022-09-27T15:51:42.6758968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3xeicls/_remote_module_non_scriptable.py
2022-09-27T15:51:42.6763095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0_pka8m
2022-09-27T15:51:42.6765055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0_pka8m/_remote_module_non_scriptable.py
2022-09-27T15:51:42.7691458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:42.7761987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf6teq8y
2022-09-27T15:51:42.7764011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf6teq8y/_remote_module_non_scriptable.py
2022-09-27T15:51:43.0779771Z ok (2.152s)
2022-09-27T15:51:43.0780295Z 
2022-09-27T15:51:43.0780814Z ----------------------------------------------------------------------
2022-09-27T15:51:43.0781069Z Ran 1 test in 2.152s
2022-09-27T15:51:43.0781187Z 
2022-09-27T15:51:43.0781249Z OK
2022-09-27T15:51:43.0781346Z 
2022-09-27T15:51:43.0781424Z Generating XML reports...
2022-09-27T15:51:43.0826756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155140.xml
2022-09-27T15:51:44.4143783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:44.4144728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:44.4153270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:44.4153909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:44.7517174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:44.7527274Z 
2022-09-27T15:51:44.7527567Z Running tests...
2022-09-27T15:51:44.7528194Z ----------------------------------------------------------------------
2022-09-27T15:51:44.7546803Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:51:44.9881299Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19484
2022-09-27T15:51:44.9908708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19485
2022-09-27T15:51:44.9936382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19486
2022-09-27T15:51:46.0704320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:46.0704711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:46.0708019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:46.0708363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:46.0800034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:46.0800816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:46.1071607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:46.1072510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:46.1074552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:46.1075160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:46.1158088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:46.1158773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:46.4452950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:46.4553928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:46.4554471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:46.4555436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:46.4556232Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:46.4556773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:46.4561242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:46.4563449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:46.4564072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:46.4630781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ydvtqf_
2022-09-27T15:51:46.4631828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ydvtqf_/_remote_module_non_scriptable.py
2022-09-27T15:51:46.4632507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2tgve2h
2022-09-27T15:51:46.4633822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mhle6wq
2022-09-27T15:51:46.4634592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2tgve2h/_remote_module_non_scriptable.py
2022-09-27T15:51:46.4636004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mhle6wq/_remote_module_non_scriptable.py
2022-09-27T15:51:46.4840850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:46.4841532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:46.4842207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:46.7982880Z ok (2.045s)
2022-09-27T15:51:46.7983117Z 
2022-09-27T15:51:46.7983648Z ----------------------------------------------------------------------
2022-09-27T15:51:46.7984154Z Ran 1 test in 2.045s
2022-09-27T15:51:46.7984259Z 
2022-09-27T15:51:46.7984321Z OK
2022-09-27T15:51:46.7984414Z 
2022-09-27T15:51:46.7984511Z Generating XML reports...
2022-09-27T15:51:46.8029022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml
2022-09-27T15:51:48.1043366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:48.1043742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:48.1051697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:48.1052289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:48.4424073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:48.4433621Z 
2022-09-27T15:51:48.4433736Z Running tests...
2022-09-27T15:51:48.4434311Z ----------------------------------------------------------------------
2022-09-27T15:51:48.4453470Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:51:48.6802661Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19537
2022-09-27T15:51:48.6829181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19538
2022-09-27T15:51:48.6855409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19539
2022-09-27T15:51:49.7640099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:49.7640753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:49.7641565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:49.7642192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:49.7988536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:49.7989208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:49.7990006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:49.7990585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:49.8305179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:49.8305883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:49.8625296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:49.8625948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:50.1959311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:50.1989268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:50.1989716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:50.1990409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:50.1991024Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:50.2060372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:50.2096517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:50.2097382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:50.2166555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp7y7mdu
2022-09-27T15:51:50.2167197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ib2067t
2022-09-27T15:51:50.2167736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp7y7mdu/_remote_module_non_scriptable.py
2022-09-27T15:51:50.2169286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ib2067t/_remote_module_non_scriptable.py
2022-09-27T15:51:50.3069473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:50.3138993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt5wxrff
2022-09-27T15:51:50.3140849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt5wxrff/_remote_module_non_scriptable.py
2022-09-27T15:51:50.3337188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:50.3337858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:50.3338524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:50.5905925Z ok (2.147s)
2022-09-27T15:51:50.5906165Z 
2022-09-27T15:51:50.5906591Z ----------------------------------------------------------------------
2022-09-27T15:51:50.5906855Z Ran 1 test in 2.147s
2022-09-27T15:51:50.5906974Z 
2022-09-27T15:51:50.5907036Z OK
2022-09-27T15:51:50.5907129Z 
2022-09-27T15:51:50.5907207Z Generating XML reports...
2022-09-27T15:51:50.5952430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155148.xml
2022-09-27T15:51:51.9055442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:51.9055831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:51.9063143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:51.9063494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:52.2433950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:52.2443880Z 
2022-09-27T15:51:52.2444335Z Running tests...
2022-09-27T15:51:52.2444740Z ----------------------------------------------------------------------
2022-09-27T15:51:52.2459066Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:51:52.4793942Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19590
2022-09-27T15:51:52.4820312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19591
2022-09-27T15:51:52.4850275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19592
2022-09-27T15:51:53.5753244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:53.5753838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:53.6101988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:53.6102374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:53.6232439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:53.6232937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:53.6371060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:53.6371487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:53.6580004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:53.6580483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:53.6712770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:53.6713140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:54.0079984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:54.0180727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:54.0181616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:54.0182222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:54.0182847Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:54.0183363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:54.0189214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:54.0189815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:54.0191297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:54.0259795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a4c1aam
2022-09-27T15:51:54.0261779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a4c1aam/_remote_module_non_scriptable.py
2022-09-27T15:51:54.0289448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6u_96bd
2022-09-27T15:51:54.0289847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07j8magx
2022-09-27T15:51:54.0291857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6u_96bd/_remote_module_non_scriptable.py
2022-09-27T15:51:54.0292325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07j8magx/_remote_module_non_scriptable.py
2022-09-27T15:51:54.3903110Z ok (2.146s)
2022-09-27T15:51:54.3903358Z 
2022-09-27T15:51:54.3904007Z ----------------------------------------------------------------------
2022-09-27T15:51:54.3904463Z Ran 1 test in 2.146s
2022-09-27T15:51:54.3904575Z 
2022-09-27T15:51:54.3904643Z OK
2022-09-27T15:51:54.3904737Z 
2022-09-27T15:51:54.3904847Z Generating XML reports...
2022-09-27T15:51:54.3952247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155152.xml
2022-09-27T15:51:55.6798825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:55.6799204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:55.6806838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:55.6807291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:56.0176683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:56.0186514Z 
2022-09-27T15:51:56.0187033Z Running tests...
2022-09-27T15:51:56.0187448Z ----------------------------------------------------------------------
2022-09-27T15:51:56.2521081Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19643
2022-09-27T15:51:56.2548331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19644
2022-09-27T15:51:56.2574852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19645
2022-09-27T15:51:57.3442644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:57.3443026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:57.3679065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:57.3679473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:57.3763512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:57.3764198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.3866176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:57.3866806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:57.4026394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:57.4026829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.4196439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:57.4197103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.7467797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:57.7567980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:51:57.7568471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:57.7569149Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:57.7569696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:57.7570224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:51:57.7674691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:51:57.7679008Z STAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8578162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:57.8579358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:57.8582695Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8585914Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8593337Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8594533Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8595065Z 
2022-09-27T15:51:57.8608532Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8615808Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8616111Z 
2022-09-27T15:51:57.8622605Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8623426Z STAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8624358Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8632831Z STAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8633654Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8634603Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:57.8640235Z STAGE:2022-09-27 15:51:57 19645:19645 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8641280Z STAGE:2022-09-27 15:51:57 19643:19643 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:57.8642043Z STAGE:2022-09-27 15:51:57 19644:19644 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:58.2625609Z ok (2.243s)
2022-09-27T15:51:58.2625856Z 
2022-09-27T15:51:58.2626399Z ----------------------------------------------------------------------
2022-09-27T15:51:58.2626662Z Ran 1 test in 2.244s
2022-09-27T15:51:58.2626780Z 
2022-09-27T15:51:58.2626828Z OK
2022-09-27T15:51:58.2626922Z 
2022-09-27T15:51:58.2627018Z Generating XML reports...
2022-09-27T15:51:58.2674945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155156.xml
2022-09-27T15:51:59.5680983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:59.5681409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:59.5688219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:59.5688600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:59.9061448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:59.9071047Z 
2022-09-27T15:51:59.9071143Z Running tests...
2022-09-27T15:51:59.9071719Z ----------------------------------------------------------------------
2022-09-27T15:52:00.1405110Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19699
2022-09-27T15:52:00.1432606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19700
2022-09-27T15:52:00.1459506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19701
2022-09-27T15:52:01.2151306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.2151988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.2215332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.2215893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.2229544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.2230009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.2529377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.2529774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:01.2587606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.2588348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:01.2597670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.2598060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:01.5966660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:01.5967367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:01.5967745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:01.5968366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:01.5968902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:01.5969572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:01.6071550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:01.6076670Z STAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.6975216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:01.6975809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:01.6979757Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.6980629Z STAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.6989561Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.6990475Z STAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.6990790Z 
2022-09-27T15:52:01.6998862Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7000024Z STAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7000606Z STAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7006685Z STAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.7007223Z 
2022-09-27T15:52:01.7015782Z STAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7016596Z STAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7019640Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.7029243Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:01.7035761Z STAGE:2022-09-27 15:52:01 19699:19699 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.7036868Z STAGE:2022-09-27 15:52:01 19700:19700 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:01 19701:19701 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:01.7037375Z 
2022-09-27T15:52:02.0512884Z ok (2.144s)
2022-09-27T15:52:02.0513134Z 
2022-09-27T15:52:02.0513605Z ----------------------------------------------------------------------
2022-09-27T15:52:02.0514003Z Ran 1 test in 2.144s
2022-09-27T15:52:02.0514193Z 
2022-09-27T15:52:02.0514303Z OK
2022-09-27T15:52:02.0514449Z 
2022-09-27T15:52:02.0514601Z Generating XML reports...
2022-09-27T15:52:02.0560888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155159.xml
2022-09-27T15:52:03.3673373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:03.3673776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:03.3680807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:03.3681266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:03.7051931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:03.7060885Z 
2022-09-27T15:52:03.7061152Z Running tests...
2022-09-27T15:52:03.7061723Z ----------------------------------------------------------------------
2022-09-27T15:52:03.9456776Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19755
2022-09-27T15:52:03.9483359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19756
2022-09-27T15:52:03.9510730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19757
2022-09-27T15:52:05.0092798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:05.0093426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:05.0252595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:05.0253313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:05.0340746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:05.0341282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:05.0439986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:05.0440570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:05.0615041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:05.0615668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:05.0688680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:05.0689368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:05.3852115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:05.3928336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:05.3928753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:05.3929386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:05.3929908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:05.3953144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:05.4035565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:05.4035960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:05.4962040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:05.5268854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:05.5269477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:05.5270078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:05.5270984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:05.5271807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:05.5272609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:05.5275520Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5375888Z STAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5376416Z STAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5383356Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5384374Z STAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5384668Z 
2022-09-27T15:52:05.5391012Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5394069Z STAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5394362Z 
2022-09-27T15:52:05.5400634Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5401272Z STAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5401723Z STAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5407519Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5409872Z STAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:05.5410301Z 
2022-09-27T15:52:05.5415264Z STAGE:2022-09-27 15:52:05 19755:19755 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5415729Z STAGE:2022-09-27 15:52:05 19756:19756 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.5416242Z STAGE:2022-09-27 15:52:05 19757:19757 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:05.9563935Z ok (2.250s)
2022-09-27T15:52:05.9564220Z 
2022-09-27T15:52:05.9564777Z ----------------------------------------------------------------------
2022-09-27T15:52:05.9565117Z Ran 1 test in 2.250s
2022-09-27T15:52:05.9565236Z 
2022-09-27T15:52:05.9565298Z OK
2022-09-27T15:52:05.9565392Z 
2022-09-27T15:52:05.9565476Z Generating XML reports...
2022-09-27T15:52:05.9611046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155203.xml
2022-09-27T15:52:07.2574494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:07.2574864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:07.2582602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:07.2583071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:07.5958046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:07.5967742Z 
2022-09-27T15:52:07.5967844Z Running tests...
2022-09-27T15:52:07.5968405Z ----------------------------------------------------------------------
2022-09-27T15:52:07.8313161Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19820
2022-09-27T15:52:07.8340209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19821
2022-09-27T15:52:07.8366862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19822
2022-09-27T15:52:08.9043601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:08.9044500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:08.9045377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:08.9046165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:08.9267108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:08.9267767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:08.9398853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:08.9399385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:08.9400518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:08.9400991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:08.9605343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:08.9605981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:09.2906685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:09.2907338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:09.2908322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:09.2908728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:09.2909232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:09.2909772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:09.3012518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:09.3917316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:09.3917937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:09.3918510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:09.4022386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:09.4023065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:09.4023695Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:09.4024383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:09.4028115Z STAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4122757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:09.4127813Z STAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4134339Z STAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:09.4134821Z STAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:09.4142047Z STAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4142653Z STAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4147757Z STAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:09.4148125Z 
2022-09-27T15:52:09.4153382Z STAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4153829Z STAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:09.4157622Z STAGE:2022-09-27 15:52:09 19822:19822 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:09 19821:19821 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:09.4157919Z 
2022-09-27T15:52:09.8416409Z ok (2.244s)
2022-09-27T15:52:09.8416624Z 
2022-09-27T15:52:09.8417016Z ----------------------------------------------------------------------
2022-09-27T15:52:09.8417313Z Ran 1 test in 2.245s
2022-09-27T15:52:09.8417436Z 
2022-09-27T15:52:09.8417514Z OK
2022-09-27T15:52:09.8417607Z 
2022-09-27T15:52:09.8417705Z Generating XML reports...
2022-09-27T15:52:09.8464300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml
2022-09-27T15:52:11.1715049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:11.1715704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:11.1724172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:11.1724812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:11.5303233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:11.5314288Z 
2022-09-27T15:52:11.5314625Z Running tests...
2022-09-27T15:52:11.5315272Z ----------------------------------------------------------------------
2022-09-27T15:52:11.7699755Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19881
2022-09-27T15:52:11.7727254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19882
2022-09-27T15:52:11.7754854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19883
2022-09-27T15:52:12.8289628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:12.8290314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:12.8382302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:12.8382673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:12.8621068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:12.8621675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:12.8723983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:12.8724618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:12.8898770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:12.8899380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:12.9216354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:12.9217005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:13.2703738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:13.2704787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:13.2705235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:13.2705942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:13.2706461Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:13.2706992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:13.2809540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:13.2813513Z STAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3714821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:13.3715441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:13.3720430Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3721103Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3728876Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3729340Z 
2022-09-27T15:52:13.3729704Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3737596Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3738150Z STAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3738595Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3745356Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3746308Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3746724Z 
2022-09-27T15:52:13.3752264Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3752907Z STAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3753606Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:13.3758917Z STAGE:2022-09-27 15:52:13 19881:19881 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3759671Z STAGE:2022-09-27 15:52:13 19882:19882 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:13 19883:19883 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:13.3759974Z 
2022-09-27T15:52:13.7802941Z ok (2.248s)
2022-09-27T15:52:13.7803182Z 
2022-09-27T15:52:13.7803602Z ----------------------------------------------------------------------
2022-09-27T15:52:13.7803856Z Ran 1 test in 2.249s
2022-09-27T15:52:13.7803962Z 
2022-09-27T15:52:13.7804026Z OK
2022-09-27T15:52:13.7804119Z 
2022-09-27T15:52:13.7804225Z Generating XML reports...
2022-09-27T15:52:13.7848890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155211.xml
2022-09-27T15:52:15.0893457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:15.0894443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:15.0902511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:15.0903325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:15.4410984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:15.4421391Z 
2022-09-27T15:52:15.4421830Z Running tests...
2022-09-27T15:52:15.4422243Z ----------------------------------------------------------------------
2022-09-27T15:52:15.6775000Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19937
2022-09-27T15:52:15.6801422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19938
2022-09-27T15:52:15.6828578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19939
2022-09-27T15:52:16.7648049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:16.7648678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:16.7725307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:16.7725947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:16.7985857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:16.7986266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:16.8071739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:16.8072187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:16.8367982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:16.8368345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:16.8683276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:16.8683658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:17.2045089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:17.2046335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:17.2046934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:17.2047817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:17.2103203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:17.2104234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:17.2104873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:17.2105437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:17.2106035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:17.2106661Z STAGE:2022-09-27 15:52:17 19939:19939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:17.2107105Z STAGE:2022-09-27 15:52:17 19937:19937 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:17.2107530Z STAGE:2022-09-27 15:52:17 19938:19938 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:17.2108130Z STAGE:2022-09-27 15:52:17 19937:19937 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:17.2108570Z STAGE:2022-09-27 15:52:17 19939:19939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:17.2109071Z STAGE:2022-09-27 15:52:17 19938:19938 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:17.5879135Z ok (2.145s)
2022-09-27T15:52:17.5879598Z 
2022-09-27T15:52:17.5880494Z ----------------------------------------------------------------------
2022-09-27T15:52:17.5880927Z Ran 1 test in 2.146s
2022-09-27T15:52:17.5881047Z 
2022-09-27T15:52:17.5881110Z OK
2022-09-27T15:52:17.5881211Z 
2022-09-27T15:52:17.5881306Z Generating XML reports...
2022-09-27T15:52:17.5925816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155215.xml
2022-09-27T15:52:18.8756700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:18.8757108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:18.8764653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:18.8765032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:19.2149458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:19.2158953Z 
2022-09-27T15:52:19.2159414Z Running tests...
2022-09-27T15:52:19.2160036Z ----------------------------------------------------------------------
2022-09-27T15:52:19.4552169Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19993
2022-09-27T15:52:19.4578139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19994
2022-09-27T15:52:19.4604980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19995
2022-09-27T15:52:20.5071542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:20.5072233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:20.5180984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:20.5181504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:20.5276781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:20.5277311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:20.5419254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:20.5419865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:20.5533158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:20.5533555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:20.5623508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:20.5624104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:20.8961393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:20.9062411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:20.9063057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:20.9064039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:20.9064785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:20.9065745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:20.9071063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:20.9072069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:20.9072568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:20.9076060Z STAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9077971Z STAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9081714Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9089639Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9090710Z STAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9091017Z 
2022-09-27T15:52:20.9106704Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9114506Z STAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9114801Z 
2022-09-27T15:52:20.9120517Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9121297Z STAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9121749Z STAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9134778Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9138811Z STAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:20.9139278Z 
2022-09-27T15:52:20.9144477Z STAGE:2022-09-27 15:52:20 19993:19993 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9145711Z STAGE:2022-09-27 15:52:20 19994:19994 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:20 19995:19995 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:20.9146252Z 
2022-09-27T15:52:21.2655548Z ok (2.049s)
2022-09-27T15:52:21.2655813Z 
2022-09-27T15:52:21.2656389Z ----------------------------------------------------------------------
2022-09-27T15:52:21.2656726Z Ran 1 test in 2.050s
2022-09-27T15:52:21.2656846Z 
2022-09-27T15:52:21.2656908Z OK
2022-09-27T15:52:21.2656987Z 
2022-09-27T15:52:21.2657091Z Generating XML reports...
2022-09-27T15:52:21.2701956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml
2022-09-27T15:52:22.5745580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:22.5745964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:22.5753889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:22.5754308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.9153326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:22.9162603Z 
2022-09-27T15:52:22.9163245Z Running tests...
2022-09-27T15:52:22.9163762Z ----------------------------------------------------------------------
2022-09-27T15:52:22.9179629Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-09-27T15:52:22.9180259Z 
2022-09-27T15:52:22.9180654Z ----------------------------------------------------------------------
2022-09-27T15:52:22.9181044Z Ran 1 test in 0.002s
2022-09-27T15:52:22.9181246Z 
2022-09-27T15:52:22.9181375Z OK (skipped=1)
2022-09-27T15:52:22.9181557Z 
2022-09-27T15:52:22.9181707Z Generating XML reports...
2022-09-27T15:52:22.9211556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155222.xml
2022-09-27T15:52:24.0935734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:24.0936118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:24.0945122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:24.0945484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:24.4399295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:24.4408851Z 
2022-09-27T15:52:24.4408958Z Running tests...
2022-09-27T15:52:24.4409411Z ----------------------------------------------------------------------
2022-09-27T15:52:24.4427236Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T15:52:24.4427632Z 
2022-09-27T15:52:24.4428078Z ----------------------------------------------------------------------
2022-09-27T15:52:24.4428398Z Ran 1 test in 0.002s
2022-09-27T15:52:24.4428512Z 
2022-09-27T15:52:24.4428587Z OK (skipped=1)
2022-09-27T15:52:24.4428698Z 
2022-09-27T15:52:24.4428785Z Generating XML reports...
2022-09-27T15:52:24.4459111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155224.xml
2022-09-27T15:52:25.6609022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:25.6609698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:25.6618229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:25.6618856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:26.0170073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:26.0180447Z 
2022-09-27T15:52:26.0180804Z Running tests...
2022-09-27T15:52:26.0181436Z ----------------------------------------------------------------------
2022-09-27T15:52:26.2576330Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20069
2022-09-27T15:52:26.2605399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20070
2022-09-27T15:52:26.2632953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20071
2022-09-27T15:52:27.3338649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:27.3339255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:27.3548300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:27.3548984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:27.3630698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:27.3631310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:27.3671282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:27.3671815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:27.3905145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:27.3905497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:27.3973033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:27.3973392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:27.7374305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:27.7374992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:27.7375585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:27.7376207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:27.7376745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:27.7377266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:27.7481073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:27.7481432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:27.8382864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:27.8591679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:27.8693607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:27.8694334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:27.8695107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:27.8695643Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:27.8696167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:27.8698748Z STAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8699231Z STAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8798618Z STAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8806602Z STAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:27.8807031Z 
2022-09-27T15:52:27.8807041Z 
2022-09-27T15:52:27.8821628Z STAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8822382Z STAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8822981Z STAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8830067Z STAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:27.8830530Z 
2022-09-27T15:52:27.8830534Z 
2022-09-27T15:52:27.8839716Z STAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8840834Z STAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:27.8841347Z 
2022-09-27T15:52:27.8845069Z STAGE:2022-09-27 15:52:27 20070:20070 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:27.8846172Z STAGE:2022-09-27 15:52:27 20071:20071 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:27 20069:20069 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:27.8846707Z 
2022-09-27T15:52:28.2685159Z ok (2.250s)
2022-09-27T15:52:28.2685385Z 
2022-09-27T15:52:28.2685886Z ----------------------------------------------------------------------
2022-09-27T15:52:28.2686367Z Ran 1 test in 2.250s
2022-09-27T15:52:28.2686559Z 
2022-09-27T15:52:28.2686671Z OK
2022-09-27T15:52:28.2686837Z 
2022-09-27T15:52:28.2686998Z Generating XML reports...
2022-09-27T15:52:28.2731334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155226.xml
2022-09-27T15:52:29.5731294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:29.5731678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:29.5740187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:29.5740557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:29.9236725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:29.9246139Z 
2022-09-27T15:52:29.9246278Z Running tests...
2022-09-27T15:52:29.9247004Z ----------------------------------------------------------------------
2022-09-27T15:52:30.1584668Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20134
2022-09-27T15:52:30.1611284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20135
2022-09-27T15:52:30.1638459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20136
2022-09-27T15:52:31.2162235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:31.2162864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:31.2499734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:31.2500355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:31.2552663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:31.2553303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:31.2554114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:31.2554746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:31.2904865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:31.2905545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:31.2906774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:31.2907348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:31.6179416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:31.6280578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:31.6281548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:31.6282168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:31.6283005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:31.6283747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:31.6289605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:31.7290953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:31.7291534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:31.7291912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:31.7400540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:31.7401094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:31.7401903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:31.7402661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:31.7406059Z STAGE:2022-09-27 15:52:31 20135:20135 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:31 20136:20136 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.7406544Z 
2022-09-27T15:52:31.7412918Z STAGE:2022-09-27 15:52:31 20136:20136 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:31.7413483Z STAGE:2022-09-27 15:52:31 20135:20135 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:31.7426832Z STAGE:2022-09-27 15:52:31 20136:20136 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.7427484Z STAGE:2022-09-27 15:52:31 20135:20135 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.7432347Z STAGE:2022-09-27 15:52:31 20135:20135 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:31 20136:20136 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:31.7432827Z 
2022-09-27T15:52:31.7496570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:32.1687857Z ok (2.244s)
2022-09-27T15:52:32.1688139Z 
2022-09-27T15:52:32.1688673Z ----------------------------------------------------------------------
2022-09-27T15:52:32.1688955Z Ran 1 test in 2.244s
2022-09-27T15:52:32.1689075Z 
2022-09-27T15:52:32.1689137Z OK
2022-09-27T15:52:32.1689231Z 
2022-09-27T15:52:32.1689333Z Generating XML reports...
2022-09-27T15:52:32.1735299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155229.xml
2022-09-27T15:52:33.4820111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:33.4820504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:33.4828860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:33.4829215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:33.8214497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:33.8224636Z 
2022-09-27T15:52:33.8224959Z Running tests...
2022-09-27T15:52:33.8225640Z ----------------------------------------------------------------------
2022-09-27T15:52:33.8243044Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:52:33.8243591Z 
2022-09-27T15:52:33.8244148Z ----------------------------------------------------------------------
2022-09-27T15:52:33.8244622Z Ran 1 test in 0.002s
2022-09-27T15:52:33.8244815Z 
2022-09-27T15:52:33.8244932Z OK (skipped=1)
2022-09-27T15:52:33.8245075Z 
2022-09-27T15:52:33.8245150Z Generating XML reports...
2022-09-27T15:52:33.8274045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155233.xml
2022-09-27T15:52:35.0362229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:35.0362638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:35.0369576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:35.0369955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:35.3734512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:35.3743466Z 
2022-09-27T15:52:35.3743566Z Running tests...
2022-09-27T15:52:35.3744228Z ----------------------------------------------------------------------
2022-09-27T15:52:35.3762171Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:52:35.3762622Z 
2022-09-27T15:52:35.3763085Z ----------------------------------------------------------------------
2022-09-27T15:52:35.3763323Z Ran 1 test in 0.002s
2022-09-27T15:52:35.3763440Z 
2022-09-27T15:52:35.3763525Z OK (skipped=1)
2022-09-27T15:52:35.3763638Z 
2022-09-27T15:52:35.3763725Z Generating XML reports...
2022-09-27T15:52:35.3793521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155235.xml
2022-09-27T15:52:36.5603393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:36.5603775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:36.5611165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:36.5611662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:36.9105077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:36.9114484Z 
2022-09-27T15:52:36.9114626Z Running tests...
2022-09-27T15:52:36.9115200Z ----------------------------------------------------------------------
2022-09-27T15:52:37.1468625Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20215
2022-09-27T15:52:37.1495937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20216
2022-09-27T15:52:37.1522546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20217
2022-09-27T15:52:38.2249971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.2250604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.2458001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.2458964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.2534095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.2534826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.2599514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.2600034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.2819401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.2820006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.2880051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.2880695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.6043152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:38.6142650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:38.6143280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:38.6144445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:38.6145406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:38.6146295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:38.6250325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:38.6250752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:38.7153015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:39.0570136Z ok (2.145s)
2022-09-27T15:52:39.0570382Z 
2022-09-27T15:52:39.0570804Z ----------------------------------------------------------------------
2022-09-27T15:52:39.0571065Z Ran 1 test in 2.145s
2022-09-27T15:52:39.0571182Z 
2022-09-27T15:52:39.0571247Z OK
2022-09-27T15:52:39.0571329Z 
2022-09-27T15:52:39.0571435Z Generating XML reports...
2022-09-27T15:52:39.0616380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155236.xml
2022-09-27T15:52:40.3859157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:40.3859524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:40.3867698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:40.3868057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:40.7254346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:40.7263391Z 
2022-09-27T15:52:40.7263715Z Running tests...
2022-09-27T15:52:40.7264303Z ----------------------------------------------------------------------
2022-09-27T15:52:40.9629400Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20268
2022-09-27T15:52:40.9655144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20269
2022-09-27T15:52:40.9682658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20270
2022-09-27T15:52:42.0353309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:42.0353689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:42.0701216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:42.0701742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:42.1008288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:42.1008632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:42.1221710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:42.1222052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:42.1342914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:42.1343280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:42.1548024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:42.1548395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:42.4707616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:42.4808324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:42.4808830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:42.4809458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:42.4809975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:42.4810511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:42.4915696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:42.4916717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:42.5818706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:42.6233464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:42.6234214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:42.6234973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:42.6235382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:42.6235898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:42.6236415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:42.6386277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T15:52:42.6484715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-09-27T15:52:42.6485181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T15:52:42.6485804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:52:42.6486341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:52:42.6488085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-09-27T15:52:42.6809982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T15:52:42.6910618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-09-27T15:52:42.6911462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T15:52:42.6912137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:52:42.6912798Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:52:42.6913328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-09-27T15:52:43.0736951Z ok (2.347s)
2022-09-27T15:52:43.0737205Z 
2022-09-27T15:52:43.0737762Z ----------------------------------------------------------------------
2022-09-27T15:52:43.0738132Z Ran 1 test in 2.347s
2022-09-27T15:52:43.0738266Z 
2022-09-27T15:52:43.0738313Z OK
2022-09-27T15:52:43.0738406Z 
2022-09-27T15:52:43.0738499Z Generating XML reports...
2022-09-27T15:52:43.0785486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155240.xml
2022-09-27T15:52:44.3737275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:44.3737701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:44.3745629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:44.3746022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:44.7118435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:44.7127695Z 
2022-09-27T15:52:44.7127838Z Running tests...
2022-09-27T15:52:44.7128301Z ----------------------------------------------------------------------
2022-09-27T15:52:44.7151327Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-09-27T15:52:44.7151680Z 
2022-09-27T15:52:44.7152123Z ----------------------------------------------------------------------
2022-09-27T15:52:44.7152520Z Ran 1 test in 0.002s
2022-09-27T15:52:44.7152636Z 
2022-09-27T15:52:44.7152709Z OK (skipped=1)
2022-09-27T15:52:44.7152820Z 
2022-09-27T15:52:44.7152892Z Generating XML reports...
2022-09-27T15:52:44.7182425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155244.xml
2022-09-27T15:52:45.9340086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:45.9340473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:45.9347897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:45.9348450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:46.2716742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:46.2726340Z 
2022-09-27T15:52:46.2726828Z Running tests...
2022-09-27T15:52:46.2727437Z ----------------------------------------------------------------------
2022-09-27T15:52:46.5066570Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20367
2022-09-27T15:52:46.5092956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20368
2022-09-27T15:52:46.5119820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20369
2022-09-27T15:52:47.5963750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:47.5964395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:47.6235237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:47.6235912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:47.6255950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:47.6256591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:47.6288247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:47.6288932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:47.6590062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:47.6590636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:47.6605787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:47.6606479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:48.0123831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:48.0224533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:48.0225304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:48.0226389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:48.0227123Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:48.0227638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:48.0331644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:48.1235469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:48.1236165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:48.1443882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:48.1444571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:48.1445009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:48.1445617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:48.1446156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:48.1446688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:48.1448720Z STAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1549368Z STAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1549847Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1557018Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1557898Z STAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1558560Z STAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1571322Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1575733Z STAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1576120Z 
2022-09-27T15:52:48.1583288Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1584567Z STAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1585098Z 
2022-09-27T15:52:48.1593190Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1596548Z STAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:48.1596859Z 
2022-09-27T15:52:48.1603834Z STAGE:2022-09-27 15:52:48 20368:20368 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:48 20369:20369 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.1604382Z 
2022-09-27T15:52:48.1604841Z STAGE:2022-09-27 15:52:48 20367:20367 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:48.5171256Z ok (2.244s)
2022-09-27T15:52:48.5171474Z 
2022-09-27T15:52:48.5171909Z ----------------------------------------------------------------------
2022-09-27T15:52:48.5172240Z Ran 1 test in 2.244s
2022-09-27T15:52:48.5172358Z 
2022-09-27T15:52:48.5172419Z OK
2022-09-27T15:52:48.5172526Z 
2022-09-27T15:52:48.5172620Z Generating XML reports...
2022-09-27T15:52:48.5218716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155246.xml
2022-09-27T15:52:49.8265297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:49.8265973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:49.8274054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:49.8274701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:50.1648749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:50.1658725Z 
2022-09-27T15:52:50.1659159Z Running tests...
2022-09-27T15:52:50.1659592Z ----------------------------------------------------------------------
2022-09-27T15:52:50.4027319Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20432
2022-09-27T15:52:50.4052766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20433
2022-09-27T15:52:50.4080512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20434
2022-09-27T15:52:51.4528472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:51.4528893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:51.4653736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:51.4654086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:51.4840306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:51.4840847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:51.4868867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:51.4869579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:51.5013179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:51.5013783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:51.5179504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:51.5180152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:51.8550164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:51.8650881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:51.8651866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:51.8652276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:51.8652777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:51.8653303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:51.8658614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:51.8659024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:51.8661156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:51.8865357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:51.8868870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:51.8869316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:51.8869906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:51.8870439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:51.8967123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:51.8970896Z STAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.8975301Z STAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.8975754Z 
2022-09-27T15:52:51.8982265Z STAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.8983244Z STAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.8983538Z 
2022-09-27T15:52:51.8996155Z STAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.9003121Z STAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.9003425Z 
2022-09-27T15:52:51.9011460Z STAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.9012171Z STAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.9012539Z 
2022-09-27T15:52:51.9019747Z STAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.9025147Z STAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:51.9025594Z 
2022-09-27T15:52:51.9032448Z STAGE:2022-09-27 15:52:51 20434:20434 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:51 20432:20432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.9033444Z STAGE:2022-09-27 15:52:51 20433:20433 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:51.9033784Z 
2022-09-27T15:52:52.3129838Z ok (2.147s)
2022-09-27T15:52:52.3130087Z 
2022-09-27T15:52:52.3130542Z ----------------------------------------------------------------------
2022-09-27T15:52:52.3130946Z Ran 1 test in 2.147s
2022-09-27T15:52:52.3131105Z 
2022-09-27T15:52:52.3131203Z OK
2022-09-27T15:52:52.3131346Z 
2022-09-27T15:52:52.3131483Z Generating XML reports...
2022-09-27T15:52:52.3177022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155250.xml
2022-09-27T15:52:53.6237411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.6237801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.6244550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.6245138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.9622824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:53.9631599Z 
2022-09-27T15:52:53.9631716Z Running tests...
2022-09-27T15:52:53.9632358Z ----------------------------------------------------------------------
2022-09-27T15:52:54.1965203Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20497
2022-09-27T15:52:54.1990702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20498
2022-09-27T15:52:54.2017075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20499
2022-09-27T15:52:55.2956414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:55.2957088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:55.3056928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:55.3057629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:55.3234683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:55.3235228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:55.3296635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:55.3297173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:55.3419235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:55.3419885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:55.3577274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:55.3577929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:55.6947960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:55.6948630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:55.6949048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:55.6949733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:55.6950259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:55.6951050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:55.6954633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:55.6956831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:55.6957670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:55.7261488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:55.7362938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:55.7363398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:55.7364133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:55.7364678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:55.7464258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:55.7469195Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7469858Z STAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7470287Z STAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7477133Z STAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7477959Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7478414Z STAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7490550Z STAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7495986Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7496402Z 
2022-09-27T15:52:55.7504089Z STAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7505095Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7505398Z 
2022-09-27T15:52:55.7514657Z STAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7517495Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7518258Z STAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:55.7523992Z STAGE:2022-09-27 15:52:55 20497:20497 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7524944Z STAGE:2022-09-27 15:52:55 20498:20498 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:55 20499:20499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:55.7525245Z 
2022-09-27T15:52:56.1068717Z ok (2.143s)
2022-09-27T15:52:56.1069077Z 
2022-09-27T15:52:56.1069623Z ----------------------------------------------------------------------
2022-09-27T15:52:56.1069919Z Ran 1 test in 2.144s
2022-09-27T15:52:56.1070038Z 
2022-09-27T15:52:56.1070109Z OK
2022-09-27T15:52:56.1070201Z 
2022-09-27T15:52:56.1070292Z Generating XML reports...
2022-09-27T15:52:56.1115636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155253.xml
2022-09-27T15:52:57.4173321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:57.4173713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:57.4181588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:57.4181945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:57.7557196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:57.7566745Z 
2022-09-27T15:52:57.7566885Z Running tests...
2022-09-27T15:52:57.7567337Z ----------------------------------------------------------------------
2022-09-27T15:52:57.9921151Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20562
2022-09-27T15:52:57.9947078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20563
2022-09-27T15:52:57.9973650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20564
2022-09-27T15:52:59.0701060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.0701432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.0756767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.0757138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.0949045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.0949559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.1046966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.1047599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.1114975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.1115534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.1285152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.1285751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.4438737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:59.4539958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:59.4540395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:52:59.4541197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:59.4541728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:59.4542331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:52:59.4547149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:59.4548099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:59.4551838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:52:59.4656435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:59.4754784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:59.4755390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:52:59.4755960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:59.4756499Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:59.4758310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:52:59.4762722Z STAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4860249Z STAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4860708Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4870530Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:59.4870955Z 
2022-09-27T15:52:59.4870966Z 
2022-09-27T15:52:59.4886194Z STAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4886881Z STAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4887400Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4894699Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:59.4895715Z STAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:59.4896244Z 
2022-09-27T15:52:59.4906656Z STAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4907284Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4907984Z STAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:59.4914637Z STAGE:2022-09-27 15:52:59 20562:20562 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:59 20563:20563 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:59.4915148Z 
2022-09-27T15:52:59.4915623Z STAGE:2022-09-27 15:52:59 20564:20564 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:59.9024366Z ok (2.145s)
2022-09-27T15:52:59.9024618Z 
2022-09-27T15:52:59.9033418Z ----------------------------------------------------------------------
2022-09-27T15:52:59.9033789Z Ran 1 test in 2.146s
2022-09-27T15:52:59.9033906Z 
2022-09-27T15:52:59.9033966Z OK
2022-09-27T15:52:59.9034118Z 
2022-09-27T15:52:59.9034213Z Generating XML reports...
2022-09-27T15:52:59.9070284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155257.xml
2022-09-27T15:53:01.1997149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:01.1997775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:01.2005934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:01.2006572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:01.5411109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:01.5421105Z 
2022-09-27T15:53:01.5421418Z Running tests...
2022-09-27T15:53:01.5422001Z ----------------------------------------------------------------------
2022-09-27T15:53:01.7800161Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20627
2022-09-27T15:53:01.7825738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20628
2022-09-27T15:53:01.7853202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20629
2022-09-27T15:53:02.8896992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:02.8897669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:02.8934112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:02.8934777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:02.9239452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:02.9240081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:02.9284056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:02.9284695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:02.9558904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:02.9559261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:02.9888964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:02.9889348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:03.3225932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:03.3325164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:03.3325881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:03.3326661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:03.3327237Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:03.3327749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:03.3433162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:03.3434236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:03.3638546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:03.3639071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:03.4336226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:03.4339304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:03.4340073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:03.4346638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:03.4347424Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:03.4351250Z STAGE:2022-09-27 15:53:03 20628:20628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:03.4352192Z STAGE:2022-09-27 15:53:03 20629:20629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:03.4359752Z STAGE:2022-09-27 15:53:03 20628:20628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:03.4360321Z STAGE:2022-09-27 15:53:03 20629:20629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:03.4373878Z STAGE:2022-09-27 15:53:03 20628:20628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:03.4374610Z STAGE:2022-09-27 15:53:03 20629:20629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:03.4380328Z STAGE:2022-09-27 15:53:03 20628:20628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:03.4380976Z STAGE:2022-09-27 15:53:03 20629:20629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:03.8906135Z ok (2.348s)
2022-09-27T15:53:03.8906395Z 
2022-09-27T15:53:03.8906868Z ----------------------------------------------------------------------
2022-09-27T15:53:03.8907282Z Ran 1 test in 2.348s
2022-09-27T15:53:03.8907470Z 
2022-09-27T15:53:03.8907552Z OK
2022-09-27T15:53:03.8907704Z 
2022-09-27T15:53:03.8907848Z Generating XML reports...
2022-09-27T15:53:03.8954176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155301.xml
2022-09-27T15:53:05.2009151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:05.2009541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:05.2016832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:05.2017183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:05.5379076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:05.5389766Z 
2022-09-27T15:53:05.5390203Z Running tests...
2022-09-27T15:53:05.5390623Z ----------------------------------------------------------------------
2022-09-27T15:53:05.7736773Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20688
2022-09-27T15:53:05.7763178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20689
2022-09-27T15:53:05.7790341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20690
2022-09-27T15:53:06.8527445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.8528019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.8858789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.8859482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.9010336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.9011265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.9191367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.9191924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.9346496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.9347116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.9537362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.9538031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:07.2851156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:07.2951692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:07.2952318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:07.2953283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:07.2953870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:07.2954663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:07.2959749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:07.2961815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:07.2962691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:07.2963395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:07.3067235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:07.3068056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:07.3068700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:07.3073123Z STAGE:2022-09-27 15:53:07 20689:20689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:07.3165351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:07.3168999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:07.3172961Z STAGE:2022-09-27 15:53:07 20690:20690 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:07.3180624Z STAGE:2022-09-27 15:53:07 20690:20690 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:07.3181136Z STAGE:2022-09-27 15:53:07 20689:20689 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:07.3194069Z STAGE:2022-09-27 15:53:07 20690:20690 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:07.3194926Z STAGE:2022-09-27 15:53:07 20689:20689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:07.3200670Z STAGE:2022-09-27 15:53:07 20689:20689 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:07 20690:20690 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:07.3200974Z 
2022-09-27T15:53:07.7841251Z ok (2.245s)
2022-09-27T15:53:07.7841467Z 
2022-09-27T15:53:07.7842272Z ----------------------------------------------------------------------
2022-09-27T15:53:07.7842618Z Ran 1 test in 2.245s
2022-09-27T15:53:07.7842722Z 
2022-09-27T15:53:07.7842788Z OK
2022-09-27T15:53:07.7842881Z 
2022-09-27T15:53:07.7842978Z Generating XML reports...
2022-09-27T15:53:07.7888331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155305.xml
2022-09-27T15:53:09.0950883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:09.0951258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:09.0958847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:09.0959345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:09.4347247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:09.4356433Z 
2022-09-27T15:53:09.4356573Z Running tests...
2022-09-27T15:53:09.4357207Z ----------------------------------------------------------------------
2022-09-27T15:53:09.6703714Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20749
2022-09-27T15:53:09.6729917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20750
2022-09-27T15:53:09.6757176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20751
2022-09-27T15:53:10.7672386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:10.7673000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:10.7707654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:10.7708306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:10.8036389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:10.8037113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:10.8058259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:10.8058911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:10.8069034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:10.8069629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:10.8368124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:10.8368758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:11.1656869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:11.1757617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:11.1758345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:11.1759330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:11.1759875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:11.1760571Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:11.1864792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:11.2769777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:11.2770221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:11.2770766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:11.2976342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:11.2976951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:11.2977773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:11.2978600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:11.2981842Z STAGE:2022-09-27 15:53:11 20750:20750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:11.2982305Z STAGE:2022-09-27 15:53:11 20751:20751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:11.2989733Z STAGE:2022-09-27 15:53:11 20751:20751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:11.2990441Z STAGE:2022-09-27 15:53:11 20750:20750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:11.3004149Z STAGE:2022-09-27 15:53:11 20750:20750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:11.3004758Z STAGE:2022-09-27 15:53:11 20751:20751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:11.3009499Z STAGE:2022-09-27 15:53:11 20751:20751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:11.3010277Z STAGE:2022-09-27 15:53:11 20750:20750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:11.3075329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:11.7812186Z ok (2.345s)
2022-09-27T15:53:11.7812472Z 
2022-09-27T15:53:11.7812990Z ----------------------------------------------------------------------
2022-09-27T15:53:11.7813275Z Ran 1 test in 2.345s
2022-09-27T15:53:11.7813393Z 
2022-09-27T15:53:11.7813455Z OK
2022-09-27T15:53:11.7813561Z 
2022-09-27T15:53:11.7814010Z Generating XML reports...
2022-09-27T15:53:11.7859230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155309.xml
2022-09-27T15:53:13.0829683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.0830054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.0837662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.0838148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:13.4249773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:13.4261520Z 
2022-09-27T15:53:13.4261815Z Running tests...
2022-09-27T15:53:13.4262486Z ----------------------------------------------------------------------
2022-09-27T15:53:13.6614348Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20810
2022-09-27T15:53:13.6640422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20811
2022-09-27T15:53:13.6667609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20812
2022-09-27T15:53:14.7022229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:14.7022896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:14.7074573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:14.7075397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:14.7361701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:14.7362378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:14.7419081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:14.7419675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:14.7851933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:14.7852341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:14.8181167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:14.8181686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:15.1551158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:15.1551887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:15.1552319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:15.1552940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:15.1553473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:15.1553998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:15.1658160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:15.1658747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:15.1763764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:15.1764155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:15.2560237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:15.2563137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:15.2564083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:15.2572193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:15.2573076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:15.2577087Z STAGE:2022-09-27 15:53:15 20811:20811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:15.2677218Z STAGE:2022-09-27 15:53:15 20812:20812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:15.2684666Z STAGE:2022-09-27 15:53:15 20812:20812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:15.2685231Z STAGE:2022-09-27 15:53:15 20811:20811 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:15.2701470Z STAGE:2022-09-27 15:53:15 20812:20812 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:15 20811:20811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:15.2701782Z 
2022-09-27T15:53:15.2707217Z STAGE:2022-09-27 15:53:15 20812:20812 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:15 20811:20811 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:15.2707640Z 
2022-09-27T15:53:15.6718365Z ok (2.245s)
2022-09-27T15:53:15.6718623Z 
2022-09-27T15:53:15.6719161Z ----------------------------------------------------------------------
2022-09-27T15:53:15.6719562Z Ran 1 test in 2.246s
2022-09-27T15:53:15.6719680Z 
2022-09-27T15:53:15.6719727Z OK
2022-09-27T15:53:15.6719821Z 
2022-09-27T15:53:15.6719914Z Generating XML reports...
2022-09-27T15:53:15.6767337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155313.xml
2022-09-27T15:53:16.9729760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:16.9730167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:16.9737882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:16.9738260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:17.3157264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:17.3166476Z 
2022-09-27T15:53:17.3166566Z Running tests...
2022-09-27T15:53:17.3167031Z ----------------------------------------------------------------------
2022-09-27T15:53:17.5498712Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20871
2022-09-27T15:53:17.5524392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20872
2022-09-27T15:53:17.5551532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20873
2022-09-27T15:53:18.5965277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:18.5965965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:18.6154628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:18.6155307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:18.6156091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:18.6156740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:18.6311865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:18.6312414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:18.6511696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:18.6512323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:18.6513119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:18.6513699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:18.9829751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:18.9830340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:18.9830718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:18.9831333Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:18.9832020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:18.9832548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:18.9936538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:18.9940734Z STAGE:2022-09-27 15:53:18 20872:20872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0837679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:19.0842834Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0937177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:19.0941890Z STAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0949427Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0949886Z 
2022-09-27T15:53:19.0950305Z STAGE:2022-09-27 15:53:19 20872:20872 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0963213Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0963850Z STAGE:2022-09-27 15:53:19 20872:20872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0964539Z STAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0971279Z STAGE:2022-09-27 15:53:19 20872:20872 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0971931Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0972231Z 
2022-09-27T15:53:19.0979847Z STAGE:2022-09-27 15:53:19 20872:20872 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0984371Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0984822Z STAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:19.0991560Z STAGE:2022-09-27 15:53:19 20871:20871 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0992305Z STAGE:2022-09-27 15:53:19 20872:20872 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:19 20873:20873 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:19.0992603Z 
2022-09-27T15:53:19.4600915Z ok (2.143s)
2022-09-27T15:53:19.4601285Z 
2022-09-27T15:53:19.4601802Z ----------------------------------------------------------------------
2022-09-27T15:53:19.4602220Z Ran 1 test in 2.143s
2022-09-27T15:53:19.4602344Z 
2022-09-27T15:53:19.4602406Z OK
2022-09-27T15:53:19.4602506Z 
2022-09-27T15:53:19.4602600Z Generating XML reports...
2022-09-27T15:53:19.4647957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml
2022-09-27T15:53:20.7616555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:20.7616942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:20.7624254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:20.7624839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:21.1013797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:21.1023318Z 
2022-09-27T15:53:21.1023465Z Running tests...
2022-09-27T15:53:21.1024203Z ----------------------------------------------------------------------
2022-09-27T15:53:21.3405019Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20927
2022-09-27T15:53:21.3429684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20928
2022-09-27T15:53:21.3458697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20929
2022-09-27T15:53:22.4143470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:22.4144296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:22.4353908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:22.4354622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:22.4472661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:22.4473204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:22.4567650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:22.4568160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:22.4708919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:22.4709535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:22.4894408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:22.4895034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:22.8123846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:22.8225197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:22.8225883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:22.8226842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:22.8227434Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:22.8228118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:22.8233627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:22.8234409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:22.8234957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:23.1509931Z ok (2.048s)
2022-09-27T15:53:23.1510138Z 
2022-09-27T15:53:23.1510531Z ----------------------------------------------------------------------
2022-09-27T15:53:23.1510837Z Ran 1 test in 2.049s
2022-09-27T15:53:23.1510955Z 
2022-09-27T15:53:23.1511018Z OK
2022-09-27T15:53:23.1511113Z 
2022-09-27T15:53:23.1511199Z Generating XML reports...
2022-09-27T15:53:23.1556171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155321.xml
2022-09-27T15:53:24.4628186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:24.4628557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:24.4636298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:24.4636648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:24.8003447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:24.8012691Z 
2022-09-27T15:53:24.8012827Z Running tests...
2022-09-27T15:53:24.8013550Z ----------------------------------------------------------------------
2022-09-27T15:53:25.0371138Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20980
2022-09-27T15:53:25.0397818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20981
2022-09-27T15:53:25.0425443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20982
2022-09-27T15:53:26.1307161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:26.1307556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:26.1424832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:26.1425412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:26.1644996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:26.1645550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:26.1765396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:26.1765767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:26.2021546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:26.2022185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:26.2340479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:26.2341175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:26.5760695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:26.5860902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:26.5861455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:26.5862231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:26.5862772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:26.5863308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:26.5970244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:26.5970659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:26.5975615Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.5976299Z STAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6873276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:26.6878314Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6885280Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6885944Z STAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6887663Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6900311Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6904957Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6905468Z 
2022-09-27T15:53:26.6912703Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6913549Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6914376Z STAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6921604Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6925898Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:26.6926221Z 
2022-09-27T15:53:26.6933162Z STAGE:2022-09-27 15:53:26 20981:20981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6933863Z STAGE:2022-09-27 15:53:26 20980:20980 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:26 20982:20982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:26.6934154Z 
2022-09-27T15:53:27.0478271Z ok (2.246s)
2022-09-27T15:53:27.0478477Z 
2022-09-27T15:53:27.0478828Z ----------------------------------------------------------------------
2022-09-27T15:53:27.0479161Z Ran 1 test in 2.246s
2022-09-27T15:53:27.0479280Z 
2022-09-27T15:53:27.0479343Z OK
2022-09-27T15:53:27.0479454Z 
2022-09-27T15:53:27.0479536Z Generating XML reports...
2022-09-27T15:53:27.0524449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml
2022-09-27T15:53:28.3534854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:28.3535252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:28.3542618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:28.3542991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:28.6935787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:28.6945197Z 
2022-09-27T15:53:28.6945341Z Running tests...
2022-09-27T15:53:28.6945810Z ----------------------------------------------------------------------
2022-09-27T15:53:28.9298260Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21036
2022-09-27T15:53:28.9324670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21037
2022-09-27T15:53:28.9351828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21038
2022-09-27T15:53:30.0478919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:30.0479498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:30.0480482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:30.0481024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:30.0826456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:30.0827092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:30.0827892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:30.0828627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:30.0946865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:30.0947509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:30.1260622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:30.1261259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:30.4608867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:30.4709761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:30.4710203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:30.4711084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:30.4711956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:30.4712517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:30.4717259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:30.4718460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:30.4719686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:30.4722064Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4724115Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4725188Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4732932Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4733382Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4733834Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4746665Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4747353Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4748046Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4755558Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4756380Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4757137Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4764902Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4769756Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4771560Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:30.4778419Z STAGE:2022-09-27 15:53:30 21036:21036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4779261Z STAGE:2022-09-27 15:53:30 21038:21038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.4780018Z STAGE:2022-09-27 15:53:30 21037:21037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:30.8402940Z ok (2.145s)
2022-09-27T15:53:30.8403221Z 
2022-09-27T15:53:30.8403680Z ----------------------------------------------------------------------
2022-09-27T15:53:30.8404061Z Ran 1 test in 2.146s
2022-09-27T15:53:30.8404243Z 
2022-09-27T15:53:30.8404344Z OK
2022-09-27T15:53:30.8404476Z 
2022-09-27T15:53:30.8404616Z Generating XML reports...
2022-09-27T15:53:30.8449824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155328.xml
2022-09-27T15:53:32.1327305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:32.1327963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:32.1335920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:32.1336318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:32.4710902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:32.4719933Z 
2022-09-27T15:53:32.4720073Z Running tests...
2022-09-27T15:53:32.4720471Z ----------------------------------------------------------------------
2022-09-27T15:53:32.7113966Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21092
2022-09-27T15:53:32.7140355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21093
2022-09-27T15:53:32.7168325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21094
2022-09-27T15:53:33.8027359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:33.8027983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:33.8096935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:33.8097609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:33.8364800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:33.8365428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:33.8441851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:33.8442228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:33.8669343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:33.8669806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:33.8987489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:33.8988013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:34.2330132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:34.2424897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:34.2425589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:34.2426203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:34.2426893Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:34.2431156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:34.2532266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:34.2532682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:34.2537917Z STAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.2538476Z STAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3441361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:34.3445860Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3455551Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3456010Z 
2022-09-27T15:53:34.3456409Z STAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3472773Z STAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3474067Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3474543Z STAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3482874Z STAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3483855Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3484175Z 
2022-09-27T15:53:34.3494380Z STAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3494965Z STAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3495499Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:34.3503500Z STAGE:2022-09-27 15:53:34 21092:21092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3504476Z STAGE:2022-09-27 15:53:34 21093:21093 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.3505314Z STAGE:2022-09-27 15:53:34 21094:21094 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:34.7221834Z ok (2.250s)
2022-09-27T15:53:34.7222094Z 
2022-09-27T15:53:34.7222601Z ----------------------------------------------------------------------
2022-09-27T15:53:34.7223095Z Ran 1 test in 2.250s
2022-09-27T15:53:34.7223314Z 
2022-09-27T15:53:34.7223411Z OK
2022-09-27T15:53:34.7223589Z 
2022-09-27T15:53:34.7223774Z Generating XML reports...
2022-09-27T15:53:34.7268120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155332.xml
2022-09-27T15:53:36.0224950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:36.0225404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:36.0233516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:36.0234065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:36.3599284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:36.3607881Z 
2022-09-27T15:53:36.3608229Z Running tests...
2022-09-27T15:53:36.3609091Z ----------------------------------------------------------------------
2022-09-27T15:53:36.5960225Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21148
2022-09-27T15:53:36.5987095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21149
2022-09-27T15:53:36.6014154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21150
2022-09-27T15:53:37.7070021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:37.7070725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:37.7123922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:37.7124469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:37.7371798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:37.7372496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:37.7416963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:37.7417525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:37.7476071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:37.7476701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:37.7702232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:37.7702871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:38.1100943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:38.1200422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:38.1201178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:38.1202045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:38.1202580Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:38.1203099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:38.1308934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:38.1309545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:38.2210930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:38.5067303Z ok (2.145s)
2022-09-27T15:53:38.5067554Z 
2022-09-27T15:53:38.5067887Z ----------------------------------------------------------------------
2022-09-27T15:53:38.5068141Z Ran 1 test in 2.146s
2022-09-27T15:53:38.5068263Z 
2022-09-27T15:53:38.5068325Z OK
2022-09-27T15:53:38.5068419Z 
2022-09-27T15:53:38.5068503Z Generating XML reports...
2022-09-27T15:53:38.5112090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155336.xml
2022-09-27T15:53:39.8167894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:39.8168311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:39.8175213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:39.8175823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:40.1807620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:40.1816874Z 
2022-09-27T15:53:40.1816962Z Running tests...
2022-09-27T15:53:40.1817624Z ----------------------------------------------------------------------
2022-09-27T15:53:40.4215374Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21201
2022-09-27T15:53:40.4243064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21202
2022-09-27T15:53:40.4270782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21203
2022-09-27T15:53:41.4983094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:41.4983736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:41.5266613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:41.5267197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:41.5310183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:41.5310592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:41.5585691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:41.5586068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:41.5777428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:41.5777827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:41.6096522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:41.6096950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:41.9543178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:41.9644043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:41.9644993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:41.9645417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:41.9645967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:41.9646567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:41.9653382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:41.9653939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:41.9654560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:41.9958797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:42.0061147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:42.0062118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:42.0063039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:42.0064310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:42.0065869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:42.0066933Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0067440Z STAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0067911Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0072995Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0073732Z STAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0074433Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0080618Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0081542Z STAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0082224Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0088918Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0089984Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0090448Z 
2022-09-27T15:53:42.0094594Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0097061Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0097666Z STAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.0105469Z STAGE:2022-09-27 15:53:42 21201:21201 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0106254Z STAGE:2022-09-27 15:53:42 21203:21203 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.0106942Z STAGE:2022-09-27 15:53:42 21202:21202 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.4463417Z ok (2.264s)
2022-09-27T15:53:42.4463599Z 
2022-09-27T15:53:42.4464106Z ----------------------------------------------------------------------
2022-09-27T15:53:42.4464443Z Ran 1 test in 2.265s
2022-09-27T15:53:42.4464564Z 
2022-09-27T15:53:42.4464628Z OK
2022-09-27T15:53:42.4464723Z 
2022-09-27T15:53:42.4464823Z Generating XML reports...
2022-09-27T15:53:42.4510651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155340.xml
2022-09-27T15:53:43.8022183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:43.8022571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:43.8030399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:43.8030815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:44.1584110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:44.1594819Z 
2022-09-27T15:53:44.1595183Z Running tests...
2022-09-27T15:53:44.1595574Z ----------------------------------------------------------------------
2022-09-27T15:53:44.4004823Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21266
2022-09-27T15:53:44.4032296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21267
2022-09-27T15:53:44.4058762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21268
2022-09-27T15:53:45.4452698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:45.4453308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:45.4760665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:45.4761289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:45.4922545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:45.4923152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:45.4992473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:45.4993137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:45.5265929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:45.5266635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:45.5336067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:45.5336704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:45.8701215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:45.8703723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:45.8704399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:45.8705147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:45.8705703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:45.8801842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:45.8810882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:45.8811680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:45.9809979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:45.9924655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:46.0025579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:46.0026024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:46.0026688Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:46.0027400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:46.0027978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:46.0031687Z STAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0130162Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0130640Z STAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0136769Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0137538Z STAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0137906Z 
2022-09-27T15:53:46.0144465Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0147971Z STAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0148278Z 
2022-09-27T15:53:46.0154576Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0155235Z STAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0155537Z 
2022-09-27T15:53:46.0160120Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0163117Z STAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0163896Z STAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:46.0170565Z STAGE:2022-09-27 15:53:46 21267:21267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0171685Z STAGE:2022-09-27 15:53:46 21268:21268 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:46 21266:21266 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:46.0172174Z 
2022-09-27T15:53:46.4111772Z ok (2.251s)
2022-09-27T15:53:46.4112013Z 
2022-09-27T15:53:46.4112461Z ----------------------------------------------------------------------
2022-09-27T15:53:46.4112741Z Ran 1 test in 2.252s
2022-09-27T15:53:46.4112908Z 
2022-09-27T15:53:46.4112972Z OK
2022-09-27T15:53:46.4113052Z 
2022-09-27T15:53:46.4113161Z Generating XML reports...
2022-09-27T15:53:46.4159561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155344.xml
2022-09-27T15:53:47.7172677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:47.7173075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:47.7181019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:47.7181534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:48.0535665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:48.0544508Z 
2022-09-27T15:53:48.0544659Z Running tests...
2022-09-27T15:53:48.0545300Z ----------------------------------------------------------------------
2022-09-27T15:53:48.2944776Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21331
2022-09-27T15:53:48.2972472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21332
2022-09-27T15:53:48.2999857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21333
2022-09-27T15:53:49.4167780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.4168402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.4171957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.4172593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.4345683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.4346272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.4519468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.4520029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.4528556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.4529120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.4684444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.4685083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.7901424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:49.8001714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:49.8002312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:49.8003278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:49.8004181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:49.8004795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:49.8109978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:49.8110372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:49.9011430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:49.9119746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:49.9221884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:49.9222596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:49.9223219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:49.9223749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:49.9224447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:49.9227353Z STAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9228166Z STAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9228970Z STAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9234258Z STAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9235217Z STAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9235511Z 
2022-09-27T15:53:49.9242049Z STAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9245639Z STAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9245929Z 
2022-09-27T15:53:49.9252443Z STAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9253387Z STAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9253678Z 
2022-09-27T15:53:49.9257724Z STAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9262202Z STAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:49.9262541Z 
2022-09-27T15:53:49.9268453Z STAGE:2022-09-27 15:53:49 21333:21333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9269249Z STAGE:2022-09-27 15:53:49 21331:21331 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:49.9269704Z STAGE:2022-09-27 15:53:49 21332:21332 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:50.3050536Z ok (2.250s)
2022-09-27T15:53:50.3050686Z 
2022-09-27T15:53:50.3051060Z ----------------------------------------------------------------------
2022-09-27T15:53:50.3051314Z Ran 1 test in 2.251s
2022-09-27T15:53:50.3051430Z 
2022-09-27T15:53:50.3051544Z OK
2022-09-27T15:53:50.3051654Z 
2022-09-27T15:53:50.3051736Z Generating XML reports...
2022-09-27T15:53:50.3097085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155348.xml
2022-09-27T15:53:51.6404549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:51.6404924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:51.6413167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:51.6413532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:51.9783014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:51.9792708Z 
2022-09-27T15:53:51.9792825Z Running tests...
2022-09-27T15:53:51.9793257Z ----------------------------------------------------------------------
2022-09-27T15:53:52.2150227Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21396
2022-09-27T15:53:52.2176575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21397
2022-09-27T15:53:52.2203671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21398
2022-09-27T15:53:53.3341866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:53.3342478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:53.3397260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:53.3397676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:53.3398109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:53.3398437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:53.3765058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:53.3765817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:53.3780224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:53.3781088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:53.3782198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:53.3782862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:53.7268742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:53.7269370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:53.7269756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:53.7270574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:53.7271450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:53.7272219Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:53.7275878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:53.7277517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:53.8276859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:53.8387116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:53.8488867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:53.8489550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:53.8490194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:53.8490733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:53.8491274Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:53.8494112Z STAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8494571Z 
2022-09-27T15:53:53.8593628Z STAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8600129Z STAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:53.8600425Z 
2022-09-27T15:53:53.8600755Z STAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:53.8608024Z STAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8608513Z STAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8609036Z STAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8615720Z STAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:53.8616348Z 
2022-09-27T15:53:53.8616355Z 
2022-09-27T15:53:53.8620780Z STAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8621812Z STAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8622630Z STAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.8629713Z STAGE:2022-09-27 15:53:53 21396:21396 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:53 21397:21397 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:53 21398:21398 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:53.8630110Z 
2022-09-27T15:53:53.8630115Z 
2022-09-27T15:53:54.2254986Z ok (2.246s)
2022-09-27T15:53:54.2255166Z 
2022-09-27T15:53:54.2255535Z ----------------------------------------------------------------------
2022-09-27T15:53:54.2255795Z Ran 1 test in 2.246s
2022-09-27T15:53:54.2255919Z 
2022-09-27T15:53:54.2255968Z OK
2022-09-27T15:53:54.2256080Z 
2022-09-27T15:53:54.2256218Z Generating XML reports...
2022-09-27T15:53:54.2301039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155351.xml
2022-09-27T15:53:55.5341338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:55.5341733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:55.5349246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:55.5349760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:55.8727303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:55.8736076Z 
2022-09-27T15:53:55.8736213Z Running tests...
2022-09-27T15:53:55.8736808Z ----------------------------------------------------------------------
2022-09-27T15:53:56.1077633Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21461
2022-09-27T15:53:56.1105293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21462
2022-09-27T15:53:56.1132064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21463
2022-09-27T15:53:57.1965487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:57.1966067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:57.2012809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:57.2013397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:57.2083028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:57.2083670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:57.2311806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:57.2312337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:57.2371446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:57.2372076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:57.2437667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:57.2438270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:57.5804840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:57.5805445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:57.5805962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:53:57.5806607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:57.5807232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:57.5807763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:53:57.5814380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:53:57.6814361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:57.6815055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:57.6817950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:57.6925071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:57.6925625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:53:57.6926253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:57.6926779Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:57.6929412Z STAGE:2022-09-27 15:53:57 21462:21462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:57.6930061Z STAGE:2022-09-27 15:53:57 21463:21463 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:57.6937137Z STAGE:2022-09-27 15:53:57 21462:21462 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:57 21463:21463 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:57.6937438Z 
2022-09-27T15:53:57.6944342Z STAGE:2022-09-27 15:53:57 21462:21462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:57.6945138Z STAGE:2022-09-27 15:53:57 21463:21463 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:57.6950039Z STAGE:2022-09-27 15:53:57 21463:21463 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:57 21462:21462 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:57.6950338Z 
2022-09-27T15:53:57.7021899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:53:58.2184195Z ok (2.344s)
2022-09-27T15:53:58.2184497Z 
2022-09-27T15:53:58.2185057Z ----------------------------------------------------------------------
2022-09-27T15:53:58.2185441Z Ran 1 test in 2.345s
2022-09-27T15:53:58.2185618Z 
2022-09-27T15:53:58.2185715Z OK
2022-09-27T15:53:58.2185863Z 
2022-09-27T15:53:58.2185985Z Generating XML reports...
2022-09-27T15:53:58.2231087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155355.xml
2022-09-27T15:53:59.5464767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:59.5465210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:59.5473072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:59.5473423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:59.8852903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:59.8861686Z 
2022-09-27T15:53:59.8861861Z Running tests...
2022-09-27T15:53:59.8862202Z ----------------------------------------------------------------------
2022-09-27T15:54:00.1212859Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21522
2022-09-27T15:54:00.1239265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21523
2022-09-27T15:54:00.1266632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21524
2022-09-27T15:54:01.1897891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:01.1898505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:01.1928604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:01.1929185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:01.2247019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:01.2247766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:01.2277709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:01.2278284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:01.2498530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:01.2499081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:01.2852528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:01.2853218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:01.6310353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:01.6411153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:01.6411730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:01.6412671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:01.6413403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:01.6414182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:01.6518309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:01.7423256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:01.7424420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:01.7425090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:01.7629818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:01.7630429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:54:01.7631342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:01.7632166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:01.7634917Z STAGE:2022-09-27 15:54:01 21524:21524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.7635400Z STAGE:2022-09-27 15:54:01 21523:21523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.7641654Z STAGE:2022-09-27 15:54:01 21524:21524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.7642357Z STAGE:2022-09-27 15:54:01 21523:21523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.7649283Z STAGE:2022-09-27 15:54:01 21524:21524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.7650249Z STAGE:2022-09-27 15:54:01 21523:21523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.7655611Z STAGE:2022-09-27 15:54:01 21524:21524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.7656338Z STAGE:2022-09-27 15:54:01 21523:21523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.7729392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:02.2318842Z ok (2.345s)
2022-09-27T15:54:02.2319125Z 
2022-09-27T15:54:02.2319570Z ----------------------------------------------------------------------
2022-09-27T15:54:02.2319824Z Ran 1 test in 2.346s
2022-09-27T15:54:02.2319939Z 
2022-09-27T15:54:02.2320001Z OK
2022-09-27T15:54:02.2320093Z 
2022-09-27T15:54:02.2320190Z Generating XML reports...
2022-09-27T15:54:02.2364402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155359.xml
2022-09-27T15:54:03.5707224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:03.5707612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:03.5716281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:03.5716682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:03.9115685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:03.9124564Z 
2022-09-27T15:54:03.9124700Z Running tests...
2022-09-27T15:54:03.9125310Z ----------------------------------------------------------------------
2022-09-27T15:54:04.1494398Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21583
2022-09-27T15:54:04.1521691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21584
2022-09-27T15:54:04.1548279Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21585
2022-09-27T15:54:05.1987843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:05.1988519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:05.2305758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:05.2306404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:05.2421894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:05.2422479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:05.2496463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:05.2497070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:05.2769196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:05.2769841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:05.2842734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:05.2843525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:05.6112134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:05.6212961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:05.6214408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:05.6215054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:05.6215551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:05.6216065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:05.6220678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:05.6221898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:05.6222269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:05.6222629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:05.6428491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:54:05.6428975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:05.6429767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:05.6430306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:05.6433094Z STAGE:2022-09-27 15:54:05 21584:21584 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:05.6526677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:05.6533674Z STAGE:2022-09-27 15:54:05 21585:21585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:05.6540606Z STAGE:2022-09-27 15:54:05 21585:21585 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:05 21584:21584 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:05.6540920Z 
2022-09-27T15:54:05.6550061Z STAGE:2022-09-27 15:54:05 21585:21585 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:05 21584:21584 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:05.6550440Z 
2022-09-27T15:54:05.6555408Z STAGE:2022-09-27 15:54:05 21585:21585 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:05.6556128Z STAGE:2022-09-27 15:54:05 21584:21584 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:06.0599451Z ok (2.147s)
2022-09-27T15:54:06.0599766Z 
2022-09-27T15:54:06.0600329Z ----------------------------------------------------------------------
2022-09-27T15:54:06.0600611Z Ran 1 test in 2.147s
2022-09-27T15:54:06.0600726Z 
2022-09-27T15:54:06.0600798Z OK
2022-09-27T15:54:06.0600890Z 
2022-09-27T15:54:06.0600971Z Generating XML reports...
2022-09-27T15:54:06.0647281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155403.xml
2022-09-27T15:54:07.3617110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:07.3617546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:07.3625748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:07.3626121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:07.6982793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:07.6991467Z 
2022-09-27T15:54:07.6991618Z Running tests...
2022-09-27T15:54:07.6992022Z ----------------------------------------------------------------------
2022-09-27T15:54:07.9368047Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21644
2022-09-27T15:54:07.9394486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21645
2022-09-27T15:54:07.9421590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21646
2022-09-27T15:54:08.9938004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:08.9938407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:09.0058132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:09.0058506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:09.0173455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:09.0173946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:09.0289625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:09.0290174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:09.0416079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:09.0416575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:09.0521263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:09.0521968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:09.3767230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:09.3796501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:09.3797191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:09.3797831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:09.3798362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:09.3868348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:09.3902864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:09.3904070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:09.4108656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:09.4109049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:54:09.4876002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:09.4878926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:09.4879624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:09.4917190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:09.4918025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:54:09.4920945Z STAGE:2022-09-27 15:54:09 21645:21645 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:09.4921670Z STAGE:2022-09-27 15:54:09 21646:21646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:09.4927488Z STAGE:2022-09-27 15:54:09 21645:21645 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:09 21646:21646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:09.4927792Z 
2022-09-27T15:54:09.4934884Z STAGE:2022-09-27 15:54:09 21645:21645 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:09 21646:21646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:09.4935276Z 
2022-09-27T15:54:09.4939925Z STAGE:2022-09-27 15:54:09 21645:21645 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:09 21646:21646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:09.4940270Z 
2022-09-27T15:54:09.9474804Z ok (2.248s)
2022-09-27T15:54:09.9475030Z 
2022-09-27T15:54:09.9475582Z ----------------------------------------------------------------------
2022-09-27T15:54:09.9475972Z Ran 1 test in 2.248s
2022-09-27T15:54:09.9476092Z 
2022-09-27T15:54:09.9476154Z OK
2022-09-27T15:54:09.9476248Z 
2022-09-27T15:54:09.9476341Z Generating XML reports...
2022-09-27T15:54:09.9519424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155407.xml
2022-09-27T15:54:11.2491882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.2492290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.2500130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.2500506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.5905332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:11.5914821Z 
2022-09-27T15:54:11.5914973Z Running tests...
2022-09-27T15:54:11.5915401Z ----------------------------------------------------------------------
2022-09-27T15:54:11.8274594Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21705
2022-09-27T15:54:11.8302195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21706
2022-09-27T15:54:11.8328986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21707
2022-09-27T15:54:12.8734879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:12.8735469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:12.8817192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:12.8817821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:12.9078969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:12.9079637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:12.9167929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:12.9168637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:12.9287820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:12.9288483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:12.9612616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:12.9613211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:13.2938471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:13.3038894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:13.3039402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:13.3040029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:13.3040548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:13.3041070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:13.3145632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:13.3150142Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4048991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:13.4049617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:13.4054696Z STAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4055221Z STAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4061615Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4062058Z STAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4062513Z STAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4069832Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4073164Z STAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4073556Z 
2022-09-27T15:54:13.4080382Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4080789Z 
2022-09-27T15:54:13.4080797Z 
2022-09-27T15:54:13.4085698Z STAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4086515Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4087218Z STAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:13.4094263Z STAGE:2022-09-27 15:54:13 21706:21706 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4095073Z STAGE:2022-09-27 15:54:13 21707:21707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.4095978Z STAGE:2022-09-27 15:54:13 21705:21705 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:13.8380341Z ok (2.246s)
2022-09-27T15:54:13.8380539Z 
2022-09-27T15:54:13.8381610Z ----------------------------------------------------------------------
2022-09-27T15:54:13.8382017Z Ran 1 test in 2.246s
2022-09-27T15:54:13.8382125Z 
2022-09-27T15:54:13.8382193Z OK
2022-09-27T15:54:13.8382295Z 
2022-09-27T15:54:13.8382389Z Generating XML reports...
2022-09-27T15:54:13.8427841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155411.xml
2022-09-27T15:54:15.1434565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:15.1435173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:15.1443021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:15.1443380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:15.4833151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:15.4842570Z 
2022-09-27T15:54:15.4842676Z Running tests...
2022-09-27T15:54:15.4843906Z ----------------------------------------------------------------------
2022-09-27T15:54:15.7185877Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21761
2022-09-27T15:54:15.7212644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21762
2022-09-27T15:54:15.7239158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21763
2022-09-27T15:54:16.7713659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:16.7714271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:16.7849339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:16.7849725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:16.8050365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:16.8050883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:16.8196984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:16.8197502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:16.8382634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:16.8383128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:16.8696523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:16.8697110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:17.2323490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:17.2424619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:17.2425485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:17.2425898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:17.2426906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:17.2427434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:17.2432669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:17.2433579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:17.2436604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:17.2437486Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2439741Z STAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2441586Z STAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2447405Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2447871Z STAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2454905Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2455651Z STAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2458646Z STAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2466508Z STAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2473365Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2474490Z STAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2474825Z 
2022-09-27T15:54:17.2478737Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2481581Z STAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2482141Z STAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:17.2490045Z STAGE:2022-09-27 15:54:17 21763:21763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2491184Z STAGE:2022-09-27 15:54:17 21761:21761 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:17 21762:21762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:17.2491686Z 
2022-09-27T15:54:17.6290535Z ok (2.144s)
2022-09-27T15:54:17.6290789Z 
2022-09-27T15:54:17.6291120Z ----------------------------------------------------------------------
2022-09-27T15:54:17.6291374Z Ran 1 test in 2.145s
2022-09-27T15:54:17.6291491Z 
2022-09-27T15:54:17.6291550Z OK
2022-09-27T15:54:17.6291630Z 
2022-09-27T15:54:17.6291723Z Generating XML reports...
2022-09-27T15:54:17.6337224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155415.xml
2022-09-27T15:54:18.9542178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.9542563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.9550395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.9550771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:19.2948144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:19.2957203Z 
2022-09-27T15:54:19.2957402Z Running tests...
2022-09-27T15:54:19.2957757Z ----------------------------------------------------------------------
2022-09-27T15:54:19.5306773Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21817
2022-09-27T15:54:19.5332593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21818
2022-09-27T15:54:19.5359563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21819
2022-09-27T15:54:20.6389582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:20.6390252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:20.6683115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:20.6683850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:20.6709111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:20.6709770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:20.6891325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:20.6891714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:20.7018767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:20.7019153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:20.7222002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:20.7222625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:21.0405788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:21.0506870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:21.0507515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:21.0508447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:21.0509014Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:21.0509763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:21.0515544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:21.0516062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:21.0517318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:21.3408623Z skip: CUDA is not available. (2.045s)
2022-09-27T15:54:21.3408893Z 
2022-09-27T15:54:21.3409393Z ----------------------------------------------------------------------
2022-09-27T15:54:21.3409819Z Ran 1 test in 2.045s
2022-09-27T15:54:21.3409943Z 
2022-09-27T15:54:21.3410016Z OK (skipped=1)
2022-09-27T15:54:21.3410129Z 
2022-09-27T15:54:21.3410215Z Generating XML reports...
2022-09-27T15:54:21.3455764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155419.xml
2022-09-27T15:54:22.6329901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:22.6330330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:22.6337951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:22.6338320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:22.9764484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:22.9773790Z 
2022-09-27T15:54:22.9774019Z Running tests...
2022-09-27T15:54:22.9774502Z ----------------------------------------------------------------------
2022-09-27T15:54:23.2250304Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21870
2022-09-27T15:54:23.2277563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21871
2022-09-27T15:54:23.2304486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21872
2022-09-27T15:54:24.2668773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:24.2669746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:24.3010789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:24.3011317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:24.3116068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:24.3116420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:24.3186853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:24.3187229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:24.3478716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:24.3479378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:24.3556324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:24.3557024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:24.6780176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:24.6879541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:24.6880131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:24.6880902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:24.6881449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:24.6881965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:24.6987085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:24.6987738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:24.7894519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:25.1354894Z skip: CUDA is not available. (2.158s)
2022-09-27T15:54:25.1355160Z 
2022-09-27T15:54:25.1355641Z ----------------------------------------------------------------------
2022-09-27T15:54:25.1356085Z Ran 1 test in 2.158s
2022-09-27T15:54:25.1356289Z 
2022-09-27T15:54:25.1356410Z OK (skipped=1)
2022-09-27T15:54:25.1356617Z 
2022-09-27T15:54:25.1356770Z Generating XML reports...
2022-09-27T15:54:25.1401607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml
2022-09-27T15:54:26.4701873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:26.4702505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:26.4710840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:26.4711466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:26.8144893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:26.8155274Z 
2022-09-27T15:54:26.8155607Z Running tests...
2022-09-27T15:54:26.8156214Z ----------------------------------------------------------------------
2022-09-27T15:54:27.0531357Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21923
2022-09-27T15:54:27.0557165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21924
2022-09-27T15:54:27.0584447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21925
2022-09-27T15:54:28.1426208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:28.1426845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:28.1452518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:28.1453124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:28.1507166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:28.1507579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:28.1783203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:28.1783618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:28.1813492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:28.1814138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:28.1869781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:28.1870155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:28.5172787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:28.5273894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:28.5287329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:28.5287978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:28.5375768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:28.5376450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:28.5394555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:28.5398945Z STAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6383788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:28.6386265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:28.6389936Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6391341Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6397822Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6398719Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6399432Z STAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6405763Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6409032Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6409335Z 
2022-09-27T15:54:28.6416341Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6416985Z 
2022-09-27T15:54:28.6417389Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6421964Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6422544Z STAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6423401Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:28.6430676Z STAGE:2022-09-27 15:54:28 21925:21925 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6431490Z STAGE:2022-09-27 15:54:28 21924:21924 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:28.6432246Z STAGE:2022-09-27 15:54:28 21923:21923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:29.0634082Z ok (2.247s)
2022-09-27T15:54:29.0634280Z 
2022-09-27T15:54:29.0634776Z ----------------------------------------------------------------------
2022-09-27T15:54:29.0635212Z Ran 1 test in 2.248s
2022-09-27T15:54:29.0635324Z 
2022-09-27T15:54:29.0635387Z OK
2022-09-27T15:54:29.0635478Z 
2022-09-27T15:54:29.0635574Z Generating XML reports...
2022-09-27T15:54:29.0680934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155426.xml
2022-09-27T15:54:30.3745178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:30.3745544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:30.3753059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:30.3753419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:30.7167500Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:30.7176407Z 
2022-09-27T15:54:30.7176670Z Running tests...
2022-09-27T15:54:30.7177316Z ----------------------------------------------------------------------
2022-09-27T15:54:30.9550486Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21979
2022-09-27T15:54:30.9576085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21980
2022-09-27T15:54:30.9603061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21981
2022-09-27T15:54:32.0641511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:32.0642090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:32.0954280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:32.0954911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:32.1050097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:32.1050484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:32.1221866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:32.1222210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:32.1383585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:32.1384786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:32.1559995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:32.1560958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:32.5073362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:32.5073825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:32.5074200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:32.5074860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:32.5075466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:32.5076045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:32.5080736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:32.5081483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:32.5083451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:32.8650493Z skip: CUDA is not available. (2.147s)
2022-09-27T15:54:32.8650823Z 
2022-09-27T15:54:32.8651277Z ----------------------------------------------------------------------
2022-09-27T15:54:32.8651521Z Ran 1 test in 2.147s
2022-09-27T15:54:32.8651639Z 
2022-09-27T15:54:32.8651712Z OK (skipped=1)
2022-09-27T15:54:32.8651820Z 
2022-09-27T15:54:32.8651911Z Generating XML reports...
2022-09-27T15:54:32.8695933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155430.xml
2022-09-27T15:54:34.1585104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:34.1585509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:34.1593219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:34.1593693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:34.4978632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:34.4987903Z 
2022-09-27T15:54:34.4988023Z Running tests...
2022-09-27T15:54:34.4989204Z ----------------------------------------------------------------------
2022-09-27T15:54:34.7346287Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22032
2022-09-27T15:54:34.7374937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22033
2022-09-27T15:54:34.7403850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22034
2022-09-27T15:54:35.7791660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:35.7792291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:35.7964212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:35.7964612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:35.8126805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:35.8127224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:35.8298720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:35.8299206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:35.8474507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:35.8474848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:35.8791878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:35.8792262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:36.2158020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:36.2158778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:36.2159175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:36.2159781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:36.2160335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:36.2160864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:36.2164329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:36.2164981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:36.2168533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:36.2169342Z STAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2170726Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2173356Z STAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2178901Z STAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2179528Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2179976Z STAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2185905Z STAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2187218Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2187791Z STAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2195658Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2196662Z STAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2197182Z 
2022-09-27T15:54:36.2202094Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2203264Z STAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.2203705Z 
2022-09-27T15:54:36.2209752Z STAGE:2022-09-27 15:54:36 22032:22032 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2210863Z STAGE:2022-09-27 15:54:36 22034:22034 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:36 22033:22033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:36.2211161Z 
2022-09-27T15:54:36.6453274Z ok (2.146s)
2022-09-27T15:54:36.6453502Z 
2022-09-27T15:54:36.6454134Z ----------------------------------------------------------------------
2022-09-27T15:54:36.6454417Z Ran 1 test in 2.146s
2022-09-27T15:54:36.6454564Z 
2022-09-27T15:54:36.6454640Z OK
2022-09-27T15:54:36.6454735Z 
2022-09-27T15:54:36.6454829Z Generating XML reports...
2022-09-27T15:54:36.6500572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155434.xml
2022-09-27T15:54:37.9505151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:37.9505568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:37.9513024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:37.9513403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:38.2896494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:38.2905935Z 
2022-09-27T15:54:38.2906046Z Running tests...
2022-09-27T15:54:38.2906613Z ----------------------------------------------------------------------
2022-09-27T15:54:38.5250571Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22088
2022-09-27T15:54:38.5277520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22089
2022-09-27T15:54:38.5304212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22090
2022-09-27T15:54:39.6226033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:39.6226659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:39.6545693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:39.6546143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:39.6607690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:39.6608250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:39.6864969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:39.6865453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:39.6933516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:39.6934069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:39.7194418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:39.7194861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.0381654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:40.0482194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:40.0482819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:40.0483819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:40.0484590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:40.0485651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:40.0588709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:40.0593164Z STAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1492732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:40.1493409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:40.1497226Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1499150Z STAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1505000Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1505538Z STAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1505981Z STAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1513036Z STAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1513645Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1514105Z STAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1520284Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1521014Z STAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1521728Z STAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1526382Z STAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1526883Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1527320Z STAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:40.1535469Z STAGE:2022-09-27 15:54:40 22088:22088 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:40 22089:22089 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:40 22090:22090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:40.1536287Z 
2022-09-27T15:54:40.1536294Z 
2022-09-27T15:54:40.5356091Z ok (2.245s)
2022-09-27T15:54:40.5356303Z 
2022-09-27T15:54:40.5356756Z ----------------------------------------------------------------------
2022-09-27T15:54:40.5357211Z Ran 1 test in 2.245s
2022-09-27T15:54:40.5357396Z 
2022-09-27T15:54:40.5357458Z OK
2022-09-27T15:54:40.5357549Z 
2022-09-27T15:54:40.5357641Z Generating XML reports...
2022-09-27T15:54:40.5402163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155438.xml
2022-09-27T15:54:41.8637981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:41.8638607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:41.8646440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:41.8647091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:42.2055602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:42.2066276Z 
2022-09-27T15:54:42.2066612Z Running tests...
2022-09-27T15:54:42.2067245Z ----------------------------------------------------------------------
2022-09-27T15:54:42.4422701Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22144
2022-09-27T15:54:42.4448412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22145
2022-09-27T15:54:42.4475945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22146
2022-09-27T15:54:43.4984783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:43.4985200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:43.5132650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:43.5132999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:43.5330467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:43.5331055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:43.5332439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:43.5332927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:43.5490000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:43.5490600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:43.5666052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:43.5666706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:43.8900692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:43.8901385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:43.8901776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:43.8902676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:43.8903611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:43.8904654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:43.8907064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:43.8908010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:43.8909783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:43.8914108Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8914630Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8915247Z STAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8921246Z STAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8922033Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8922752Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8930023Z STAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8930556Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8931283Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8938351Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8939071Z STAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8939504Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8943719Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8947146Z STAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8947784Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:43.8954813Z STAGE:2022-09-27 15:54:43 22145:22145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8955445Z STAGE:2022-09-27 15:54:43 22146:22146 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:43 22144:22144 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:43.8955744Z 
2022-09-27T15:54:44.2526018Z ok (2.046s)
2022-09-27T15:54:44.2526277Z 
2022-09-27T15:54:44.2526770Z ----------------------------------------------------------------------
2022-09-27T15:54:44.2527037Z Ran 1 test in 2.046s
2022-09-27T15:54:44.2527156Z 
2022-09-27T15:54:44.2527218Z OK
2022-09-27T15:54:44.2527312Z 
2022-09-27T15:54:44.2527398Z Generating XML reports...
2022-09-27T15:54:44.2571560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155442.xml
2022-09-27T15:54:45.5406635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:45.5407017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:45.5414657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:45.5415206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:45.8788618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:45.8797815Z 
2022-09-27T15:54:45.8797966Z Running tests...
2022-09-27T15:54:45.8798514Z ----------------------------------------------------------------------
2022-09-27T15:54:46.1156392Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22200
2022-09-27T15:54:46.1182998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22201
2022-09-27T15:54:46.1211944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22202
2022-09-27T15:54:47.2304728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.2305406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.2520771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.2521314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.2631765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.2632142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.2638231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.2638741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.2875564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.2876434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.2978734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.6261414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.6261784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:47.6362116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:47.6362604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:47.6363538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:47.6364090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:47.6364631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:47.6370904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:47.6372898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:47.6373617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:47.9259742Z skip: CUDA is not available. (2.046s)
2022-09-27T15:54:47.9260000Z 
2022-09-27T15:54:47.9260357Z ----------------------------------------------------------------------
2022-09-27T15:54:47.9260617Z Ran 1 test in 2.046s
2022-09-27T15:54:47.9260733Z 
2022-09-27T15:54:47.9260792Z OK (skipped=1)
2022-09-27T15:54:47.9260904Z 
2022-09-27T15:54:47.9260993Z Generating XML reports...
2022-09-27T15:54:47.9307069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155445.xml
2022-09-27T15:54:49.2145864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:49.2146236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:49.2153739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:49.2154084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:49.5527004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:49.5536162Z 
2022-09-27T15:54:49.5536301Z Running tests...
2022-09-27T15:54:49.5536703Z ----------------------------------------------------------------------
2022-09-27T15:54:49.7879331Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22253
2022-09-27T15:54:49.7905565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22254
2022-09-27T15:54:49.7932139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22255
2022-09-27T15:54:50.8678926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:50.8679538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:50.8877187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:50.8877783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:50.8911636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:50.8912023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:50.9022512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:50.9023416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:50.9242109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:50.9243057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:50.9268787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:50.9269431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:51.2645113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:51.2721364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:51.2721783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:51.2722428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:51.2722950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:51.2746261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:51.2829562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:51.2844956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:51.3758022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:51.6983527Z skip: CUDA is not available. (2.144s)
2022-09-27T15:54:51.6984797Z 
2022-09-27T15:54:51.6985131Z ----------------------------------------------------------------------
2022-09-27T15:54:51.6985381Z Ran 1 test in 2.145s
2022-09-27T15:54:51.6985505Z 
2022-09-27T15:54:51.6985589Z OK (skipped=1)
2022-09-27T15:54:51.6985731Z 
2022-09-27T15:54:51.6985820Z Generating XML reports...
2022-09-27T15:54:51.7030558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155449.xml
2022-09-27T15:54:53.0246726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.0247168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.0254526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.0254902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.3682614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:53.3691718Z 
2022-09-27T15:54:53.3691858Z Running tests...
2022-09-27T15:54:53.3692289Z ----------------------------------------------------------------------
2022-09-27T15:54:53.6041685Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22306
2022-09-27T15:54:53.6068032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22307
2022-09-27T15:54:53.6095499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22308
2022-09-27T15:54:54.6910345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:54.6910736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:54.7230278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:54.7230778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:54.7232840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:54.7233276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:54.7557639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:54.7558136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:54.7702639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:54.7703108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:54.8014296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:54.8014904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:55.1208742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:55.1271282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:55.1271772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:55.1272372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:55.1272912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:55.1308940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:54:55.1378352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:55.1379428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:55.2321039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:55.5147062Z skip: CUDA is not available. (2.145s)
2022-09-27T15:54:55.5147255Z 
2022-09-27T15:54:55.5148122Z ----------------------------------------------------------------------
2022-09-27T15:54:55.5148585Z Ran 1 test in 2.145s
2022-09-27T15:54:55.5148737Z 
2022-09-27T15:54:55.5148854Z OK (skipped=1)
2022-09-27T15:54:55.5149023Z 
2022-09-27T15:54:55.5149126Z Generating XML reports...
2022-09-27T15:54:55.5194098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155453.xml
2022-09-27T15:54:56.8414579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:56.8415207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:56.8423347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:56.8424221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:57.1805820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:57.1815939Z 
2022-09-27T15:54:57.1816273Z Running tests...
2022-09-27T15:54:57.1832127Z ----------------------------------------------------------------------
2022-09-27T15:54:57.1832680Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:54:57.1833061Z 
2022-09-27T15:54:57.1833444Z ----------------------------------------------------------------------
2022-09-27T15:54:57.1833875Z Ran 1 test in 0.002s
2022-09-27T15:54:57.1834076Z 
2022-09-27T15:54:57.1834191Z OK (skipped=1)
2022-09-27T15:54:57.1834382Z 
2022-09-27T15:54:57.1834534Z Generating XML reports...
2022-09-27T15:54:57.1863725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155457.xml
2022-09-27T15:54:58.3727096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:58.3727519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:58.3734772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:58.3735144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:58.7125240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:58.7134892Z 
2022-09-27T15:54:58.7135140Z Running tests...
2022-09-27T15:54:58.7135524Z ----------------------------------------------------------------------
2022-09-27T15:54:58.7152037Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:54:58.7152461Z 
2022-09-27T15:54:58.7152915Z ----------------------------------------------------------------------
2022-09-27T15:54:58.7153165Z Ran 1 test in 0.002s
2022-09-27T15:54:58.7153280Z 
2022-09-27T15:54:58.7153353Z OK (skipped=1)
2022-09-27T15:54:58.7153448Z 
2022-09-27T15:54:58.7153542Z Generating XML reports...
2022-09-27T15:54:58.7182325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155458.xml
2022-09-27T15:54:59.9006889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:59.9007525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:59.9015117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:59.9015769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:00.2382498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:00.2392958Z 
2022-09-27T15:55:00.2393460Z Running tests...
2022-09-27T15:55:00.2393935Z ----------------------------------------------------------------------
2022-09-27T15:55:00.2409895Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-09-27T15:55:00.2410279Z 
2022-09-27T15:55:00.2410698Z ----------------------------------------------------------------------
2022-09-27T15:55:00.2411102Z Ran 1 test in 0.002s
2022-09-27T15:55:00.2411224Z 
2022-09-27T15:55:00.2411299Z OK (skipped=1)
2022-09-27T15:55:00.2411409Z 
2022-09-27T15:55:00.2411495Z Generating XML reports...
2022-09-27T15:55:00.2440846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155500.xml
2022-09-27T15:55:01.4313661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:01.4314093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:01.4321762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:01.4322132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:01.7699371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:01.7709333Z 
2022-09-27T15:55:01.7709471Z Running tests...
2022-09-27T15:55:01.7709902Z ----------------------------------------------------------------------
2022-09-27T15:55:01.7726651Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-09-27T15:55:01.7727118Z 
2022-09-27T15:55:01.7727590Z ----------------------------------------------------------------------
2022-09-27T15:55:01.7727869Z Ran 1 test in 0.002s
2022-09-27T15:55:01.7727986Z 
2022-09-27T15:55:01.7728062Z OK (skipped=1)
2022-09-27T15:55:01.7728171Z 
2022-09-27T15:55:01.7728258Z Generating XML reports...
2022-09-27T15:55:01.7757577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155501.xml
2022-09-27T15:55:03.0071286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:03.0071679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:03.0078948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:03.0079403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:03.3477889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:03.3487172Z 
2022-09-27T15:55:03.3487271Z Running tests...
2022-09-27T15:55:03.3487990Z ----------------------------------------------------------------------
2022-09-27T15:55:03.3503297Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:55:03.3503651Z 
2022-09-27T15:55:03.3504190Z ----------------------------------------------------------------------
2022-09-27T15:55:03.3504456Z Ran 1 test in 0.002s
2022-09-27T15:55:03.3504571Z 
2022-09-27T15:55:03.3504654Z OK (skipped=1)
2022-09-27T15:55:03.3504826Z 
2022-09-27T15:55:03.3504917Z Generating XML reports...
2022-09-27T15:55:03.3534522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155503.xml
2022-09-27T15:55:04.5375071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:04.5375458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:04.5383587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:04.5384176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:04.8771828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:04.8781401Z 
2022-09-27T15:55:04.8781748Z Running tests...
2022-09-27T15:55:04.8798380Z ----------------------------------------------------------------------
2022-09-27T15:55:04.8798853Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:55:04.8799190Z 
2022-09-27T15:55:04.8799527Z ----------------------------------------------------------------------
2022-09-27T15:55:04.8799983Z Ran 1 test in 0.002s
2022-09-27T15:55:04.8800188Z 
2022-09-27T15:55:04.8800270Z OK (skipped=1)
2022-09-27T15:55:04.8800384Z 
2022-09-27T15:55:04.8800469Z Generating XML reports...
2022-09-27T15:55:04.8829701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155504.xml
2022-09-27T15:55:06.0729770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:06.0730134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:06.0737341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:06.0737774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:06.4101668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:06.4110996Z 
2022-09-27T15:55:06.4111114Z Running tests...
2022-09-27T15:55:06.4111906Z ----------------------------------------------------------------------
2022-09-27T15:55:06.4126939Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-09-27T15:55:06.4127282Z 
2022-09-27T15:55:06.4127712Z ----------------------------------------------------------------------
2022-09-27T15:55:06.4128407Z Ran 1 test in 0.002s
2022-09-27T15:55:06.4128540Z 
2022-09-27T15:55:06.4128620Z OK (skipped=1)
2022-09-27T15:55:06.4128729Z 
2022-09-27T15:55:06.4128816Z Generating XML reports...
2022-09-27T15:55:06.4158060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155506.xml
2022-09-27T15:55:07.6103356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:07.6103716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:07.6111842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:07.6112230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:07.9493864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:07.9503004Z 
2022-09-27T15:55:07.9503115Z Running tests...
2022-09-27T15:55:07.9503589Z ----------------------------------------------------------------------
2022-09-27T15:55:07.9521616Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:07.9522002Z 
2022-09-27T15:55:07.9522274Z ----------------------------------------------------------------------
2022-09-27T15:55:07.9522512Z Ran 1 test in 0.002s
2022-09-27T15:55:07.9522670Z 
2022-09-27T15:55:07.9522777Z OK (skipped=1)
2022-09-27T15:55:07.9522891Z 
2022-09-27T15:55:07.9522978Z Generating XML reports...
2022-09-27T15:55:07.9552460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155507.xml
2022-09-27T15:55:09.1395365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:09.1395730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:09.1403260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:09.1403634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:09.4777085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:09.4786463Z 
2022-09-27T15:55:09.4786580Z Running tests...
2022-09-27T15:55:09.4787152Z ----------------------------------------------------------------------
2022-09-27T15:55:09.4803088Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:55:09.4803394Z 
2022-09-27T15:55:09.4803634Z ----------------------------------------------------------------------
2022-09-27T15:55:09.4803936Z Ran 1 test in 0.002s
2022-09-27T15:55:09.4804052Z 
2022-09-27T15:55:09.4804126Z OK (skipped=1)
2022-09-27T15:55:09.4804235Z 
2022-09-27T15:55:09.4804323Z Generating XML reports...
2022-09-27T15:55:09.4833757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml
2022-09-27T15:55:10.7154478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:10.7154911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:10.7162902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:10.7163423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.0804064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:11.0813892Z 
2022-09-27T15:55:11.0814040Z Running tests...
2022-09-27T15:55:11.0814638Z ----------------------------------------------------------------------
2022-09-27T15:55:11.0832255Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:11.0832726Z 
2022-09-27T15:55:11.0833126Z ----------------------------------------------------------------------
2022-09-27T15:55:11.0833735Z Ran 1 test in 0.002s
2022-09-27T15:55:11.0833940Z 
2022-09-27T15:55:11.0834056Z OK (skipped=1)
2022-09-27T15:55:11.0834252Z 
2022-09-27T15:55:11.0834388Z Generating XML reports...
2022-09-27T15:55:11.0866166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155511.xml
2022-09-27T15:55:12.3168865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:12.3169539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:12.3178240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:12.3178917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:12.6559660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:12.6569792Z 
2022-09-27T15:55:12.6570135Z Running tests...
2022-09-27T15:55:12.6570752Z ----------------------------------------------------------------------
2022-09-27T15:55:12.6587515Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:12.6587920Z 
2022-09-27T15:55:12.6588322Z ----------------------------------------------------------------------
2022-09-27T15:55:12.6588753Z Ran 1 test in 0.002s
2022-09-27T15:55:12.6588896Z 
2022-09-27T15:55:12.6588976Z OK (skipped=1)
2022-09-27T15:55:12.6589087Z 
2022-09-27T15:55:12.6589173Z Generating XML reports...
2022-09-27T15:55:12.6620498Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml
2022-09-27T15:55:13.8539110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:13.8539722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:13.8548306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:13.8548976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:14.1947063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:14.1956970Z 
2022-09-27T15:55:14.1957075Z Running tests...
2022-09-27T15:55:14.1957490Z ----------------------------------------------------------------------
2022-09-27T15:55:14.1974347Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:55:14.1974709Z 
2022-09-27T15:55:14.1974956Z ----------------------------------------------------------------------
2022-09-27T15:55:14.1975200Z Ran 1 test in 0.002s
2022-09-27T15:55:14.1975342Z 
2022-09-27T15:55:14.1975450Z OK (skipped=1)
2022-09-27T15:55:14.1975562Z 
2022-09-27T15:55:14.1975645Z Generating XML reports...
2022-09-27T15:55:14.2005326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml
2022-09-27T15:55:15.3907951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:15.3908338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:15.3916235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:15.3916609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:15.7293582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:15.7302894Z 
2022-09-27T15:55:15.7303259Z Running tests...
2022-09-27T15:55:15.7303677Z ----------------------------------------------------------------------
2022-09-27T15:55:15.7319771Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:55:15.7320503Z 
2022-09-27T15:55:15.7320777Z ----------------------------------------------------------------------
2022-09-27T15:55:15.7321028Z Ran 1 test in 0.002s
2022-09-27T15:55:15.7321144Z 
2022-09-27T15:55:15.7321204Z OK (skipped=1)
2022-09-27T15:55:15.7321314Z 
2022-09-27T15:55:15.7321400Z Generating XML reports...
2022-09-27T15:55:15.7350030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155515.xml
2022-09-27T15:55:16.9317890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:16.9318304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:16.9326073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:16.9326502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:17.2739033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:17.2748643Z 
2022-09-27T15:55:17.2748770Z Running tests...
2022-09-27T15:55:17.2749481Z ----------------------------------------------------------------------
2022-09-27T15:55:17.2766661Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:17.2767003Z 
2022-09-27T15:55:17.2767216Z ----------------------------------------------------------------------
2022-09-27T15:55:17.2767499Z Ran 1 test in 0.002s
2022-09-27T15:55:17.2767641Z 
2022-09-27T15:55:17.2767715Z OK (skipped=1)
2022-09-27T15:55:17.2767822Z 
2022-09-27T15:55:17.2767901Z Generating XML reports...
2022-09-27T15:55:17.2797878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155517.xml
2022-09-27T15:55:18.4886853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:18.4887255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:18.4894384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:18.4894752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:18.8280610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:18.8290042Z 
2022-09-27T15:55:18.8290142Z Running tests...
2022-09-27T15:55:18.8290644Z ----------------------------------------------------------------------
2022-09-27T15:55:18.8307511Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:18.8308041Z 
2022-09-27T15:55:18.8308390Z ----------------------------------------------------------------------
2022-09-27T15:55:18.8308725Z Ran 1 test in 0.002s
2022-09-27T15:55:18.8308840Z 
2022-09-27T15:55:18.8308918Z OK (skipped=1)
2022-09-27T15:55:18.8309014Z 
2022-09-27T15:55:18.8309102Z Generating XML reports...
2022-09-27T15:55:18.8338344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155518.xml
2022-09-27T15:55:20.0427975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:20.0435928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:20.0436448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:20.0437016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:20.3816926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:20.3826759Z 
2022-09-27T15:55:20.3827289Z Running tests...
2022-09-27T15:55:20.3827691Z ----------------------------------------------------------------------
2022-09-27T15:55:20.3843694Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:55:20.3844111Z 
2022-09-27T15:55:20.3844372Z ----------------------------------------------------------------------
2022-09-27T15:55:20.3844622Z Ran 1 test in 0.002s
2022-09-27T15:55:20.3844741Z 
2022-09-27T15:55:20.3844813Z OK (skipped=1)
2022-09-27T15:55:20.3844928Z 
2022-09-27T15:55:20.3844999Z Generating XML reports...
2022-09-27T15:55:20.3875100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155520.xml
2022-09-27T15:55:21.5755342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:21.5755740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:21.5763502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:21.5764001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:21.9142672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:21.9151814Z 
2022-09-27T15:55:21.9151956Z Running tests...
2022-09-27T15:55:21.9152559Z ----------------------------------------------------------------------
2022-09-27T15:55:21.9167798Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:55:21.9168131Z 
2022-09-27T15:55:21.9168451Z ----------------------------------------------------------------------
2022-09-27T15:55:21.9168709Z Ran 1 test in 0.002s
2022-09-27T15:55:21.9168823Z 
2022-09-27T15:55:21.9168891Z OK (skipped=1)
2022-09-27T15:55:21.9169000Z 
2022-09-27T15:55:21.9169144Z Generating XML reports...
2022-09-27T15:55:21.9199033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155521.xml
2022-09-27T15:55:23.1177221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:23.1177589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:23.1185689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:23.1186050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:23.4570118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:23.4579416Z 
2022-09-27T15:55:23.4579568Z Running tests...
2022-09-27T15:55:23.4579965Z ----------------------------------------------------------------------
2022-09-27T15:55:23.4596413Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:23.4596783Z 
2022-09-27T15:55:23.4597201Z ----------------------------------------------------------------------
2022-09-27T15:55:23.4597628Z Ran 1 test in 0.002s
2022-09-27T15:55:23.4597749Z 
2022-09-27T15:55:23.4597825Z OK (skipped=1)
2022-09-27T15:55:23.4597932Z 
2022-09-27T15:55:23.4598018Z Generating XML reports...
2022-09-27T15:55:23.4627435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155523.xml
2022-09-27T15:55:24.6530482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:24.6530857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:24.6537598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:24.6538078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:24.9916476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:24.9925937Z 
2022-09-27T15:55:24.9926156Z Running tests...
2022-09-27T15:55:24.9926750Z ----------------------------------------------------------------------
2022-09-27T15:55:24.9943219Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:24.9943642Z 
2022-09-27T15:55:24.9944163Z ----------------------------------------------------------------------
2022-09-27T15:55:24.9944595Z Ran 1 test in 0.002s
2022-09-27T15:55:24.9944839Z 
2022-09-27T15:55:24.9944976Z OK (skipped=1)
2022-09-27T15:55:24.9945088Z 
2022-09-27T15:55:24.9945175Z Generating XML reports...
2022-09-27T15:55:24.9974082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155524.xml
2022-09-27T15:55:26.2069625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:26.2070019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:26.2077689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:26.2078061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:26.5445199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:26.5454537Z 
2022-09-27T15:55:26.5454621Z Running tests...
2022-09-27T15:55:26.5455411Z ----------------------------------------------------------------------
2022-09-27T15:55:26.5472295Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:26.5472736Z 
2022-09-27T15:55:26.5473160Z ----------------------------------------------------------------------
2022-09-27T15:55:26.5473585Z Ran 1 test in 0.002s
2022-09-27T15:55:26.5473701Z 
2022-09-27T15:55:26.5473760Z OK (skipped=1)
2022-09-27T15:55:26.5473869Z 
2022-09-27T15:55:26.5473957Z Generating XML reports...
2022-09-27T15:55:26.5502521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155526.xml
2022-09-27T15:55:27.7394654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:27.7395035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:27.7403332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:27.7403836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:28.0784691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:28.0793800Z 
2022-09-27T15:55:28.0794088Z Running tests...
2022-09-27T15:55:28.0794530Z ----------------------------------------------------------------------
2022-09-27T15:55:28.0809188Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:55:28.0809443Z 
2022-09-27T15:55:28.0809663Z ----------------------------------------------------------------------
2022-09-27T15:55:28.0810087Z Ran 1 test in 0.002s
2022-09-27T15:55:28.0810297Z 
2022-09-27T15:55:28.0810432Z OK (skipped=1)
2022-09-27T15:55:28.0810593Z 
2022-09-27T15:55:28.0810680Z Generating XML reports...
2022-09-27T15:55:28.0840567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155528.xml
2022-09-27T15:55:29.2712263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:29.2712922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:29.2720011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:29.2720524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:29.6112040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:29.6121912Z 
2022-09-27T15:55:29.6122046Z Running tests...
2022-09-27T15:55:29.6122630Z ----------------------------------------------------------------------
2022-09-27T15:55:29.6139299Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-09-27T15:55:29.6139798Z 
2022-09-27T15:55:29.6140238Z ----------------------------------------------------------------------
2022-09-27T15:55:29.6140497Z Ran 1 test in 0.002s
2022-09-27T15:55:29.6140600Z 
2022-09-27T15:55:29.6140671Z OK (skipped=1)
2022-09-27T15:55:29.6140777Z 
2022-09-27T15:55:29.6140862Z Generating XML reports...
2022-09-27T15:55:29.6170168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155529.xml
2022-09-27T15:55:30.8045523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:30.8045894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:30.8054117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:30.8054503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:31.1445189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:31.1454700Z 
2022-09-27T15:55:31.1454873Z Running tests...
2022-09-27T15:55:31.1455488Z ----------------------------------------------------------------------
2022-09-27T15:55:31.1472089Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-09-27T15:55:31.1472502Z 
2022-09-27T15:55:31.1472779Z ----------------------------------------------------------------------
2022-09-27T15:55:31.1473025Z Ran 1 test in 0.002s
2022-09-27T15:55:31.1473142Z 
2022-09-27T15:55:31.1473216Z OK (skipped=1)
2022-09-27T15:55:31.1473313Z 
2022-09-27T15:55:31.1473400Z Generating XML reports...
2022-09-27T15:55:31.1503285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155531.xml
2022-09-27T15:55:32.3452862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:32.3453239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:32.3460957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:32.3461306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:32.6844637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:32.6854472Z 
2022-09-27T15:55:32.6854584Z Running tests...
2022-09-27T15:55:32.6855305Z ----------------------------------------------------------------------
2022-09-27T15:55:32.6873464Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:32.6873792Z 
2022-09-27T15:55:32.6874328Z ----------------------------------------------------------------------
2022-09-27T15:55:32.6874579Z Ran 1 test in 0.002s
2022-09-27T15:55:32.6874680Z 
2022-09-27T15:55:32.6874755Z OK (skipped=1)
2022-09-27T15:55:32.6874861Z 
2022-09-27T15:55:32.6875035Z Generating XML reports...
2022-09-27T15:55:32.6904555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155532.xml
2022-09-27T15:55:33.8856302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:33.8856682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:33.8864079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:33.8864462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.2235959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:34.2244719Z 
2022-09-27T15:55:34.2244843Z Running tests...
2022-09-27T15:55:34.2245450Z ----------------------------------------------------------------------
2022-09-27T15:55:34.4627178Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22599
2022-09-27T15:55:34.4653088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22600
2022-09-27T15:55:34.4680346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22601
2022-09-27T15:55:35.5528201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:35.5528586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:35.5694873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:35.5695443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:35.5861255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:35.5861795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:35.6031487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:35.6032150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:35.6317765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:35.6318401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:35.6638870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:35.6639530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:36.0189668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:36.0291901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:36.0292628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:36.0293104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:36.0293879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:36.0294684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:36.0299548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:36.0300800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:36.1302851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:36.3729197Z skip: Need at least 2 CUDA devices (2.148s)
2022-09-27T15:55:36.3729795Z 
2022-09-27T15:55:36.3730108Z ----------------------------------------------------------------------
2022-09-27T15:55:36.3730374Z Ran 1 test in 2.148s
2022-09-27T15:55:36.3730490Z 
2022-09-27T15:55:36.3730563Z OK (skipped=1)
2022-09-27T15:55:36.3730670Z 
2022-09-27T15:55:36.3730743Z Generating XML reports...
2022-09-27T15:55:36.3777647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155534.xml
2022-09-27T15:55:37.7051648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:37.7052040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:37.7059503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:37.7059876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:38.0430621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:38.0439528Z 
2022-09-27T15:55:38.0439663Z Running tests...
2022-09-27T15:55:38.0440234Z ----------------------------------------------------------------------
2022-09-27T15:55:38.0457723Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:55:38.0457993Z 
2022-09-27T15:55:38.0458211Z ----------------------------------------------------------------------
2022-09-27T15:55:38.0458460Z Ran 1 test in 0.002s
2022-09-27T15:55:38.0458563Z 
2022-09-27T15:55:38.0458637Z OK (skipped=1)
2022-09-27T15:55:38.0458745Z 
2022-09-27T15:55:38.0458830Z Generating XML reports...
2022-09-27T15:55:38.0487492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155538.xml
2022-09-27T15:55:39.2342887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:39.2343292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:39.2351121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:39.2351489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:39.5764269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:39.5773851Z 
2022-09-27T15:55:39.5773949Z Running tests...
2022-09-27T15:55:39.5774313Z ----------------------------------------------------------------------
2022-09-27T15:55:39.5790996Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:55:39.5791439Z 
2022-09-27T15:55:39.5791818Z ----------------------------------------------------------------------
2022-09-27T15:55:39.5792197Z Ran 1 test in 0.002s
2022-09-27T15:55:39.5792375Z 
2022-09-27T15:55:39.5792437Z OK (skipped=1)
2022-09-27T15:55:39.5792548Z 
2022-09-27T15:55:39.5792632Z Generating XML reports...
2022-09-27T15:55:39.5821801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155539.xml
2022-09-27T15:55:40.7631675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:40.7632055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:40.7639061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:40.7639454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.1020803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:41.1029851Z 
2022-09-27T15:55:41.1029971Z Running tests...
2022-09-27T15:55:41.1030506Z ----------------------------------------------------------------------
2022-09-27T15:55:41.3410174Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22672
2022-09-27T15:55:41.3437634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22673
2022-09-27T15:55:41.3464575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22674
2022-09-27T15:55:42.4037081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:42.4037711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:42.4118961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:42.4119601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:42.4375104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:42.4375744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:42.4462069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:42.4462702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:42.4537830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:42.4538466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:42.4851947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:42.4852601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:42.8088517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:42.8189142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:42.8189844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:42.8190791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:42.8191330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:42.8191852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:42.8196649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:42.8198174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:42.8198680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:44.4528172Z ok (3.349s)
2022-09-27T15:55:44.4528381Z 
2022-09-27T15:55:44.4528740Z ----------------------------------------------------------------------
2022-09-27T15:55:44.4528992Z Ran 1 test in 3.350s
2022-09-27T15:55:44.4529106Z 
2022-09-27T15:55:44.4529167Z OK
2022-09-27T15:55:44.4529246Z 
2022-09-27T15:55:44.4529339Z Generating XML reports...
2022-09-27T15:55:44.4573874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155541.xml
2022-09-27T15:55:45.7545077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:45.7545455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:45.7553448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:45.7553836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:46.0931578Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:46.0941700Z 
2022-09-27T15:55:46.0942147Z Running tests...
2022-09-27T15:55:46.0942525Z ----------------------------------------------------------------------
2022-09-27T15:55:46.3296614Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22725
2022-09-27T15:55:46.3323340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22726
2022-09-27T15:55:46.3349904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22727
2022-09-27T15:55:47.3873461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:47.3874107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:47.4069982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:47.4070687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:47.4206087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:47.4206634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:47.4399855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:47.4400491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:47.4613246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:47.4613901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:47.4923699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:47.4924368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:47.8190626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:47.8291630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:47.8292482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:47.8293213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:47.8293744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:47.8294276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:47.8399252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:47.8400230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:47.9305450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:48.2401322Z skip: CUDA is not available. (2.146s)
2022-09-27T15:55:48.2401643Z 
2022-09-27T15:55:48.2401965Z ----------------------------------------------------------------------
2022-09-27T15:55:48.2402239Z Ran 1 test in 2.146s
2022-09-27T15:55:48.2402353Z 
2022-09-27T15:55:48.2402426Z OK (skipped=1)
2022-09-27T15:55:48.2402533Z 
2022-09-27T15:55:48.2402619Z Generating XML reports...
2022-09-27T15:55:48.2450233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155546.xml
2022-09-27T15:55:49.5589876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:49.5590511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:49.5598024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:49.5598376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:49.8968859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:49.8977903Z 
2022-09-27T15:55:49.8978191Z Running tests...
2022-09-27T15:55:49.8978782Z ----------------------------------------------------------------------
2022-09-27T15:55:50.1322947Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22778
2022-09-27T15:55:50.1350644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22779
2022-09-27T15:55:50.1377738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22780
2022-09-27T15:55:51.2070566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.2071261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.2215759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.2216353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.2420408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.2421004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.2580911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.2581561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.2650644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.2651301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.2975356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.2976031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.6346246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:51.6447877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:51.6448394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:51.6449048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:51.6449588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:51.6450126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:51.6456332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:51.6457134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:51.6459478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:51.6763057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:55:51.6865731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:55:51.6866254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:55:51.6866922Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:55:51.6867455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:55:51.6966393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:55:53.3444539Z ok (3.446s)
2022-09-27T15:55:53.3444743Z 
2022-09-27T15:55:53.3445193Z ----------------------------------------------------------------------
2022-09-27T15:55:53.3445505Z Ran 1 test in 3.447s
2022-09-27T15:55:53.3445623Z 
2022-09-27T15:55:53.3445670Z OK
2022-09-27T15:55:53.3445779Z 
2022-09-27T15:55:53.3445907Z Generating XML reports...
2022-09-27T15:55:53.3491452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml
2022-09-27T15:55:54.6659491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.6659900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.6667911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.6668271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.0038384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:55.0047695Z 
2022-09-27T15:55:55.0047828Z Running tests...
2022-09-27T15:55:55.0048237Z ----------------------------------------------------------------------
2022-09-27T15:55:55.2436749Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22840
2022-09-27T15:55:55.2461740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22841
2022-09-27T15:55:55.2488844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22842
2022-09-27T15:55:56.3989488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:56.3990139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:56.4121293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:56.4121776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:56.4133718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:56.4134055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:56.4333016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:56.4333388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:56.4476218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:56.4476836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:56.4488838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:56.4489277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:56.7753263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:56.7853811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:56.7854396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:56.7855027Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:56.7855630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:56.7856152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:55:56.7960005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:56.8866457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:56.8866909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:57.1539494Z skip: CUDA is not available. (2.149s)
2022-09-27T15:55:57.1539774Z 
2022-09-27T15:55:57.1540324Z ----------------------------------------------------------------------
2022-09-27T15:55:57.1540677Z Ran 1 test in 2.149s
2022-09-27T15:55:57.1540794Z 
2022-09-27T15:55:57.1540874Z OK (skipped=1)
2022-09-27T15:55:57.1540969Z 
2022-09-27T15:55:57.1541054Z Generating XML reports...
2022-09-27T15:55:57.1593191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155555.xml
2022-09-27T15:55:58.4615437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:58.4616092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:58.4624276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:58.4624648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:58.7997040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:58.8005811Z 
2022-09-27T15:55:58.8005890Z Running tests...
2022-09-27T15:55:58.8006616Z ----------------------------------------------------------------------
2022-09-27T15:55:59.0349148Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22893
2022-09-27T15:55:59.0376082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22894
2022-09-27T15:55:59.0403560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22895
2022-09-27T15:56:00.1249611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:00.1250282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:00.1572491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:00.1572896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:00.1645759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:00.1646417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:00.1955939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:00.1956566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:00.2386188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:00.2386763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:00.2721694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:00.2722293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:00.6046569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:00.6147833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:00.6148688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:00.6149598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:00.6150520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:00.6151135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:00.6254399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:00.7158670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:00.7159113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:00.7160399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:00.7365219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:00.7365812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:56:00.7366502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:00.7367229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:00.7464793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:01.9469182Z ok (3.146s)
2022-09-27T15:56:01.9469365Z 
2022-09-27T15:56:01.9469819Z ----------------------------------------------------------------------
2022-09-27T15:56:01.9470102Z Ran 1 test in 3.146s
2022-09-27T15:56:01.9470220Z 
2022-09-27T15:56:01.9470285Z OK
2022-09-27T15:56:01.9470378Z 
2022-09-27T15:56:01.9470460Z Generating XML reports...
2022-09-27T15:56:01.9517281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155558.xml
2022-09-27T15:56:03.2722709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:03.2723097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:03.2730694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:03.2731077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:03.6120147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:03.6129649Z 
2022-09-27T15:56:03.6130107Z Running tests...
2022-09-27T15:56:03.6130514Z ----------------------------------------------------------------------
2022-09-27T15:56:03.8485940Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22952
2022-09-27T15:56:03.8511869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22953
2022-09-27T15:56:03.8538518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22954
2022-09-27T15:56:04.9573650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.9574277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.9879592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.9880333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.9891188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:05.0123392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:05.0137458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:05.0138234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:05.0234250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:05.0234878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:05.0462426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:05.0463120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:05.3704521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:05.3805576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:05.3806219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:05.3806858Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:05.3807391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:05.3807913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:05.3911407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:05.4818627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:05.4819172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:05.7586520Z skip: CUDA is not available. (2.145s)
2022-09-27T15:56:05.7586730Z 
2022-09-27T15:56:05.7587050Z ----------------------------------------------------------------------
2022-09-27T15:56:05.7587367Z Ran 1 test in 2.146s
2022-09-27T15:56:05.7587484Z 
2022-09-27T15:56:05.7587560Z OK (skipped=1)
2022-09-27T15:56:05.7587671Z 
2022-09-27T15:56:05.7587743Z Generating XML reports...
2022-09-27T15:56:05.7634660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155603.xml
2022-09-27T15:56:07.0651941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:07.0652348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:07.0659601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:07.0659971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:07.4051875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:07.4061304Z 
2022-09-27T15:56:07.4061399Z Running tests...
2022-09-27T15:56:07.4062417Z ----------------------------------------------------------------------
2022-09-27T15:56:07.6441012Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23005
2022-09-27T15:56:07.6468750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23006
2022-09-27T15:56:07.6495177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23007
2022-09-27T15:56:08.7202963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.7203358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.7505672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.7506048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.7770630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.7771044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.7795599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.7795935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.8118279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.8142140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.8142986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.8143590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:09.1562184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:09.1562815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:09.1563176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:09.1563824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:09.1564370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:09.1564902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:09.1668642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:09.2573101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:09.2574477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:09.2779523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:09.2881891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:56:09.2882478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:09.2883223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:09.2883957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:09.2884786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:10.6560339Z ok (3.249s)
2022-09-27T15:56:10.6560595Z 
2022-09-27T15:56:10.6561110Z ----------------------------------------------------------------------
2022-09-27T15:56:10.6561380Z Ran 1 test in 3.250s
2022-09-27T15:56:10.6561499Z 
2022-09-27T15:56:10.6561547Z OK
2022-09-27T15:56:10.6561639Z 
2022-09-27T15:56:10.6561737Z Generating XML reports...
2022-09-27T15:56:10.6606206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155607.xml
2022-09-27T15:56:12.0001691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:12.0002070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:12.0009986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:12.0010338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:12.3383517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:12.3393246Z 
2022-09-27T15:56:12.3393354Z Running tests...
2022-09-27T15:56:12.3393785Z ----------------------------------------------------------------------
2022-09-27T15:56:12.5731395Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23067
2022-09-27T15:56:12.5757601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23068
2022-09-27T15:56:12.5784436Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23069
2022-09-27T15:56:13.6654807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.6655441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.6771735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.6772412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.6992825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.6993361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.7126862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.7127402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.7256058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.7256692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.7574828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.7575213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:14.0884610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:14.0885096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:14.0885469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:14.0886122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:14.0886660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:14.0887173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:14.0990842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:14.1895403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:14.1896220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:14.2423205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:14.2504293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:14.2505403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:14.2506077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:14.2506695Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:14.2524892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:15.6852154Z ok (3.345s)
2022-09-27T15:56:15.6852420Z 
2022-09-27T15:56:15.6852888Z ----------------------------------------------------------------------
2022-09-27T15:56:15.6853273Z Ran 1 test in 3.346s
2022-09-27T15:56:15.6853446Z 
2022-09-27T15:56:15.6853550Z OK
2022-09-27T15:56:15.6853706Z 
2022-09-27T15:56:15.6853863Z Generating XML reports...
2022-09-27T15:56:15.6901118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155612.xml
2022-09-27T15:56:16.9967346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:16.9967745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:16.9974748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:16.9975101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:17.3355908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:17.3381963Z 
2022-09-27T15:56:17.3382670Z Running tests...
2022-09-27T15:56:17.3383273Z ----------------------------------------------------------------------
2022-09-27T15:56:17.5713179Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23129
2022-09-27T15:56:17.5739879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23130
2022-09-27T15:56:17.5767410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23131
2022-09-27T15:56:18.6776891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:18.6777517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:18.6999052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:18.6999695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:18.7108463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:18.7108987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:18.7157792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:18.7158177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:18.7348325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:18.7348886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:18.7493546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:19.0714467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:19.0715141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:19.0715787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:19.0716158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:19.0716911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:19.0717453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:19.0718045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:19.0820887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:19.0821265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:19.1026247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:56:19.1026643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:19.1722377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:19.1724943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:19.1726113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:19.1733890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:19.1734567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:24.4885914Z ok (7.151s)
2022-09-27T15:56:24.4886178Z 
2022-09-27T15:56:24.4886669Z ----------------------------------------------------------------------
2022-09-27T15:56:24.4886943Z Ran 1 test in 7.152s
2022-09-27T15:56:24.4887062Z 
2022-09-27T15:56:24.4887109Z OK
2022-09-27T15:56:24.4887208Z 
2022-09-27T15:56:24.4887304Z Generating XML reports...
2022-09-27T15:56:24.4931928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155617.xml
2022-09-27T15:56:25.7978447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:25.7979100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:25.7987784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:25.7988437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:26.1379904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:26.1389673Z 
2022-09-27T15:56:26.1390016Z Running tests...
2022-09-27T15:56:26.1390618Z ----------------------------------------------------------------------
2022-09-27T15:56:26.3756493Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23188
2022-09-27T15:56:26.3782997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23189
2022-09-27T15:56:26.3810673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23190
2022-09-27T15:56:27.4317974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:27.4318655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:27.4501935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:27.4502492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:27.4651181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:27.4651795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:27.4795544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:27.4796201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:27.4835490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:27.4836133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:27.5118939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:27.5119618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:27.8500054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:27.8500516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:27.8500901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:27.8501595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:27.8502197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:27.8502765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:27.8508731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:27.8509319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:27.8509990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:28.2860772Z ok (2.147s)
2022-09-27T15:56:28.2860942Z 
2022-09-27T15:56:28.2861364Z ----------------------------------------------------------------------
2022-09-27T15:56:28.2861675Z Ran 1 test in 2.147s
2022-09-27T15:56:28.2861793Z 
2022-09-27T15:56:28.2861855Z OK
2022-09-27T15:56:28.2861959Z 
2022-09-27T15:56:28.2862049Z Generating XML reports...
2022-09-27T15:56:28.2907544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155626.xml
2022-09-27T15:56:29.6295476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:29.6295867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:29.6303556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:29.6304330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:29.9675148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:29.9683888Z 
2022-09-27T15:56:29.9684034Z Running tests...
2022-09-27T15:56:29.9684664Z ----------------------------------------------------------------------
2022-09-27T15:56:30.2043131Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23241
2022-09-27T15:56:30.2069533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23242
2022-09-27T15:56:30.2097440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23243
2022-09-27T15:56:31.3202817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:31.3203444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:31.3327029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:31.3327913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:31.3350143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:31.3350723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:31.3554552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:31.3554968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:31.3685433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:31.3685849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:31.3705463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:31.3706017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:31.6950679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:31.6979959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:31.6980395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:31.6981008Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:31.6981545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:31.7052185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:31.7087316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:31.7088293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:31.8060837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:32.1148992Z ok (2.146s)
2022-09-27T15:56:32.1149245Z 
2022-09-27T15:56:32.1149705Z ----------------------------------------------------------------------
2022-09-27T15:56:32.1149946Z Ran 1 test in 2.146s
2022-09-27T15:56:32.1150063Z 
2022-09-27T15:56:32.1150123Z OK
2022-09-27T15:56:32.1150213Z 
2022-09-27T15:56:32.1150307Z Generating XML reports...
2022-09-27T15:56:32.1195419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155629.xml
2022-09-27T15:56:33.4586900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:33.4587268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:33.4595075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:33.4595516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:33.7981836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:33.7991520Z 
2022-09-27T15:56:33.7991665Z Running tests...
2022-09-27T15:56:33.7992259Z ----------------------------------------------------------------------
2022-09-27T15:56:33.8012965Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:33.8013397Z 
2022-09-27T15:56:33.8013785Z ----------------------------------------------------------------------
2022-09-27T15:56:33.8014185Z Ran 1 test in 0.002s
2022-09-27T15:56:33.8014377Z 
2022-09-27T15:56:33.8014502Z OK (skipped=1)
2022-09-27T15:56:33.8014692Z 
2022-09-27T15:56:33.8014827Z Generating XML reports...
2022-09-27T15:56:33.8045057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml
2022-09-27T15:56:34.9931538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:34.9932452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:34.9940521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:34.9941153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:35.3328288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:35.3338446Z 
2022-09-27T15:56:35.3338930Z Running tests...
2022-09-27T15:56:35.3339358Z ----------------------------------------------------------------------
2022-09-27T15:56:35.3365120Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:35.3365462Z 
2022-09-27T15:56:35.3365753Z ----------------------------------------------------------------------
2022-09-27T15:56:35.3366005Z Ran 1 test in 0.003s
2022-09-27T15:56:35.3366122Z 
2022-09-27T15:56:35.3366204Z OK (skipped=1)
2022-09-27T15:56:35.3366302Z 
2022-09-27T15:56:35.3366389Z Generating XML reports...
2022-09-27T15:56:35.3395783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155635.xml
2022-09-27T15:56:36.5498980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:36.5499359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:36.5507268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:36.5507836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:36.8882238Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:36.8891785Z 
2022-09-27T15:56:36.8891937Z Running tests...
2022-09-27T15:56:36.8892375Z ----------------------------------------------------------------------
2022-09-27T15:56:36.8915151Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:36.8915600Z 
2022-09-27T15:56:36.8915960Z ----------------------------------------------------------------------
2022-09-27T15:56:36.8916212Z Ran 1 test in 0.002s
2022-09-27T15:56:36.8916331Z 
2022-09-27T15:56:36.8916406Z OK (skipped=1)
2022-09-27T15:56:36.8916515Z 
2022-09-27T15:56:36.8916601Z Generating XML reports...
2022-09-27T15:56:36.8945733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155636.xml
2022-09-27T15:56:38.1082969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:38.1083362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:38.1091235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:38.1091616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:38.4481421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:38.4491744Z 
2022-09-27T15:56:38.4492060Z Running tests...
2022-09-27T15:56:38.4492665Z ----------------------------------------------------------------------
2022-09-27T15:56:38.4512389Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:38.4512767Z 
2022-09-27T15:56:38.4513206Z ----------------------------------------------------------------------
2022-09-27T15:56:38.4513605Z Ran 1 test in 0.002s
2022-09-27T15:56:38.4513720Z 
2022-09-27T15:56:38.4513791Z OK (skipped=1)
2022-09-27T15:56:38.4514117Z 
2022-09-27T15:56:38.4514204Z Generating XML reports...
2022-09-27T15:56:38.4543191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155638.xml
2022-09-27T15:56:39.6662935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.6663337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.6671270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.6671664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:40.0051191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:40.0060978Z 
2022-09-27T15:56:40.0061235Z Running tests...
2022-09-27T15:56:40.0061625Z ----------------------------------------------------------------------
2022-09-27T15:56:40.0079177Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:40.0079587Z 
2022-09-27T15:56:40.0080040Z ----------------------------------------------------------------------
2022-09-27T15:56:40.0080277Z Ran 1 test in 0.002s
2022-09-27T15:56:40.0080392Z 
2022-09-27T15:56:40.0080466Z OK (skipped=1)
2022-09-27T15:56:40.0080576Z 
2022-09-27T15:56:40.0080662Z Generating XML reports...
2022-09-27T15:56:40.0109589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155640.xml
2022-09-27T15:56:41.1937139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:41.1937535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:41.1945590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:41.1945980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:41.5336483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:41.5345054Z 
2022-09-27T15:56:41.5345166Z Running tests...
2022-09-27T15:56:41.5345648Z ----------------------------------------------------------------------
2022-09-27T15:56:41.5367181Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:41.5367595Z 
2022-09-27T15:56:41.5367989Z ----------------------------------------------------------------------
2022-09-27T15:56:41.5368410Z Ran 1 test in 0.002s
2022-09-27T15:56:41.5368546Z 
2022-09-27T15:56:41.5368621Z OK (skipped=1)
2022-09-27T15:56:41.5368729Z 
2022-09-27T15:56:41.5368803Z Generating XML reports...
2022-09-27T15:56:41.5398868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155641.xml
2022-09-27T15:56:42.7304186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:42.7304569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:42.7312075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:42.7312439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:43.0690173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:43.0698910Z 
2022-09-27T15:56:43.0698988Z Running tests...
2022-09-27T15:56:43.0699402Z ----------------------------------------------------------------------
2022-09-27T15:56:43.0723134Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:43.0723480Z 
2022-09-27T15:56:43.0724209Z ----------------------------------------------------------------------
2022-09-27T15:56:43.0724472Z Ran 1 test in 0.002s
2022-09-27T15:56:43.0724590Z 
2022-09-27T15:56:43.0724665Z OK (skipped=1)
2022-09-27T15:56:43.0724846Z 
2022-09-27T15:56:43.0724946Z Generating XML reports...
2022-09-27T15:56:43.0755102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml
2022-09-27T15:56:44.2668201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:44.2668847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:44.2677423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:44.2678067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:44.6072974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:44.6082816Z 
2022-09-27T15:56:44.6082922Z Running tests...
2022-09-27T15:56:44.6083386Z ----------------------------------------------------------------------
2022-09-27T15:56:44.6101826Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:56:44.6102246Z 
2022-09-27T15:56:44.6102664Z ----------------------------------------------------------------------
2022-09-27T15:56:44.6103071Z Ran 1 test in 0.002s
2022-09-27T15:56:44.6103237Z 
2022-09-27T15:56:44.6103353Z OK (skipped=1)
2022-09-27T15:56:44.6103527Z 
2022-09-27T15:56:44.6103673Z Generating XML reports...
2022-09-27T15:56:44.6134344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155644.xml
2022-09-27T15:56:45.8177630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:45.8178281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:45.8186308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:45.8186978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.1553471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:46.1563625Z 
2022-09-27T15:56:46.1564090Z Running tests...
2022-09-27T15:56:46.1564489Z ----------------------------------------------------------------------
2022-09-27T15:56:46.3914444Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23374
2022-09-27T15:56:46.3940698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23375
2022-09-27T15:56:46.3967475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23376
2022-09-27T15:56:47.4537982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:47.4538548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:47.4646227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:47.4646915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:47.4879192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:47.4879745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:47.4937299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:47.4937902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:47.4995280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:47.4995919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:47.5263620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:47.5264860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:47.8451085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:47.8551819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:47.8552399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:47.8553104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:47.8553869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:47.8554750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:47.8657981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:47.8662526Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9563598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:47.9563999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:47.9567683Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9568594Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9573371Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9574007Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9574463Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9579858Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9586277Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9587081Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9590322Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9591486Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9592216Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9597323Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9599821Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9601992Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9603261Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9604969Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9606461Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9609351Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9612793Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9615065Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9618091Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9618821Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9619580Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9623234Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9626476Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9626911Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9629417Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9630117Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9631184Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9635678Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9636808Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9638000Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9640556Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9641473Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9641914Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9645390Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9649472Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9650150Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9652577Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9653280Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9653721Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9659005Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9659313Z 
2022-09-27T15:56:47.9661425Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9661905Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9664319Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9666456Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9669179Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9669462Z 
2022-09-27T15:56:47.9672626Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9676103Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9677215Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9678468Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9680619Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9682682Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9683337Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9687040Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9687909Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9688203Z 
2022-09-27T15:56:47.9691982Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9693001Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9695527Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9696529Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9699108Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9699634Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9703164Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9703613Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9706258Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9709998Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9710490Z 
2022-09-27T15:56:47.9710947Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9713610Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9716031Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9717863Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9721648Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9722187Z 
2022-09-27T15:56:47.9722662Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9726321Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9727423Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9728316Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9730736Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9732325Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9732634Z 
2022-09-27T15:56:47.9737313Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9738014Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9741022Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9744626Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9745641Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9746391Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9748690Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9750553Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9753757Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9754847Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9758698Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9759312Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9759896Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9763949Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9766859Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9770139Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9770867Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9771310Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9774433Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9779063Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:47.9779401Z 
2022-09-27T15:56:47.9785019Z STAGE:2022-09-27 15:56:47 23376:23376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9785750Z STAGE:2022-09-27 15:56:47 23374:23374 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:47.9786179Z STAGE:2022-09-27 15:56:47 23375:23375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:48.4016401Z ok (2.245s)
2022-09-27T15:56:48.4017147Z 
2022-09-27T15:56:48.4017614Z ----------------------------------------------------------------------
2022-09-27T15:56:48.4018087Z Ran 1 test in 2.245s
2022-09-27T15:56:48.4018294Z 
2022-09-27T15:56:48.4018386Z OK
2022-09-27T15:56:48.4018465Z 
2022-09-27T15:56:48.4018559Z Generating XML reports...
2022-09-27T15:56:48.4063294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155646.xml
2022-09-27T15:56:49.7063066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:49.7063432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:49.7071190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:49.7071567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:50.0458603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:50.0468315Z 
2022-09-27T15:56:50.0468432Z Running tests...
2022-09-27T15:56:50.0469043Z ----------------------------------------------------------------------
2022-09-27T15:56:50.2736650Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.226s)
2022-09-27T15:56:50.2737497Z 
2022-09-27T15:56:50.2737865Z ----------------------------------------------------------------------
2022-09-27T15:56:50.2738330Z Ran 1 test in 0.227s
2022-09-27T15:56:50.2738512Z 
2022-09-27T15:56:50.2738631Z OK (skipped=1)
2022-09-27T15:56:50.2738815Z 
2022-09-27T15:56:50.2738958Z Generating XML reports...
2022-09-27T15:56:50.2761200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml
2022-09-27T15:56:51.5478445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:51.5479054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:51.5487320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:51.5487981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:51.8880993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:51.8890208Z 
2022-09-27T15:56:51.8890596Z Running tests...
2022-09-27T15:56:51.8891247Z ----------------------------------------------------------------------
2022-09-27T15:56:52.1283036Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23440
2022-09-27T15:56:52.1308838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23441
2022-09-27T15:56:52.1335450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23442
2022-09-27T15:56:53.2069061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:53.2069441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:53.2123718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:53.2124305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:53.2412964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:53.2413603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:53.2424551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:53.2425304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:53.2475071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:53.2475765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:53.2756012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:53.2756631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:53.6097638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:53.6198104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:53.6198688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:53.6199309Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:53.6199829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:53.6200351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:53.6306443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:53.6306925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:53.7211710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:53.7418778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:53.7520607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:56:53.7521166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:53.7521869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:53.7522458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:53.7522975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:53.7525830Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7526706Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7531800Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7543367Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7625930Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7631279Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7631589Z 
2022-09-27T15:56:53.7637356Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7641497Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7642352Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7646665Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7647402Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7652119Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7652642Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7653420Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7657250Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7658559Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7659104Z 
2022-09-27T15:56:53.7662063Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7667072Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7667391Z 
2022-09-27T15:56:53.7670633Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7671440Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7672212Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7675911Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7677665Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7678746Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7682445Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7683202Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7683968Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7687738Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7690539Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7691027Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7693713Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7694356Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7694654Z 
2022-09-27T15:56:53.7698121Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7702464Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7702755Z 
2022-09-27T15:56:53.7706130Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7706910Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7707669Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7711259Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7713922Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7714412Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7717733Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7718043Z 
2022-09-27T15:56:53.7718300Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7723104Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7724028Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7726164Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7727251Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7729827Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7734044Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7734718Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7736490Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7742558Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7745791Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7746347Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7747582Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7750923Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7753233Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7754032Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7757351Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7758492Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7758978Z 
2022-09-27T15:56:53.7762926Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7764121Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7765177Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7768953Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7769742Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7770523Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7773955Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7776104Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7777007Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7780557Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7781342Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7782147Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7785416Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7787446Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7788910Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7790092Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7798452Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7799115Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7799837Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7800261Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7800725Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7801710Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7806792Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7807494Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7809080Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7810726Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7813822Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7816196Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7816826Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7817122Z 
2022-09-27T15:56:53.7822581Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7823156Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7823794Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7827631Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7828140Z 
2022-09-27T15:56:53.7828602Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7831796Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7833176Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7835386Z STAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:53.7840057Z STAGE:2022-09-27 15:56:53 23440:23440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:53 23442:23442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:53.7840570Z 
2022-09-27T15:56:53.7840994Z STAGE:2022-09-27 15:56:53 23441:23441 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:54.2387493Z ok (2.349s)
2022-09-27T15:56:54.2387822Z 
2022-09-27T15:56:54.2388368Z ----------------------------------------------------------------------
2022-09-27T15:56:54.2388742Z Ran 1 test in 2.350s
2022-09-27T15:56:54.2388847Z 
2022-09-27T15:56:54.2388908Z OK
2022-09-27T15:56:54.2389000Z 
2022-09-27T15:56:54.2389095Z Generating XML reports...
2022-09-27T15:56:54.2435086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155651.xml
2022-09-27T15:56:55.5487863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:55.5488844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:55.5497419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:55.5498022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:55.8868832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:55.8878812Z 
2022-09-27T15:56:55.8879242Z Running tests...
2022-09-27T15:56:55.8879826Z ----------------------------------------------------------------------
2022-09-27T15:56:56.1257478Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23505
2022-09-27T15:56:56.1283024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23506
2022-09-27T15:56:56.1311589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23507
2022-09-27T15:56:57.2013374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:57.2014070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:57.2175607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:57.2175978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:57.2343625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:57.2344300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:57.2509360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:57.2509852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:57.2786464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:57.2786959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:57.3117851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:57.3118383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:57.6646127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:57.6646646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:57.6647290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:57.6647914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:56:57.6648757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:57.6649646Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:56:57.6655121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:57.6655577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:57.6655937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:57.6658573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:57.6863480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:56:57.6864256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:57.6864975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:57.6865517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:57.6866029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:56:57.6867952Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.6967821Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.6972615Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.6973085Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.6978959Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.6984098Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.6988253Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.6988737Z 
2022-09-27T15:56:57.6992818Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.6999221Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7002672Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7002997Z 
2022-09-27T15:56:57.7007966Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7010937Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7013560Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7014045Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7017478Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7021395Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7024108Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7024945Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7029162Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7029722Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7032710Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7033145Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7037097Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7039296Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7042339Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7042978Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7046730Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7048905Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7051603Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7052073Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7055936Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7058686Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7061757Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7062565Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7066554Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7067967Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7071154Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7071620Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7074852Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7077535Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7080886Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7081344Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7086283Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7088552Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:57.7091988Z STAGE:2022-09-27 15:56:57 23507:23507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:57.7092574Z STAGE:2022-09-27 15:56:57 23506:23506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:58.1361252Z ok (2.248s)
2022-09-27T15:56:58.1361906Z 
2022-09-27T15:56:58.1362945Z ----------------------------------------------------------------------
2022-09-27T15:56:58.1363298Z Ran 1 test in 2.248s
2022-09-27T15:56:58.1363417Z 
2022-09-27T15:56:58.1363492Z OK
2022-09-27T15:56:58.1363584Z 
2022-09-27T15:56:58.1363666Z Generating XML reports...
2022-09-27T15:56:58.1408580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml
2022-09-27T15:56:59.4455890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:59.4456557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:59.4465389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:59.4466020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:59.7854282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:59.7864362Z 
2022-09-27T15:56:59.7864952Z Running tests...
2022-09-27T15:56:59.7865700Z ----------------------------------------------------------------------
2022-09-27T15:57:00.0213298Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23566
2022-09-27T15:57:00.0240830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23567
2022-09-27T15:57:00.0267702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23568
2022-09-27T15:57:01.1082159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.1082755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.1136902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.1137539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.1439351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.1440139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.1497947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.1498570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.1516767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.1517401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.1854917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.1855309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.5392719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:01.5393507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:01.5393894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:01.5394524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:01.5395044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:01.5395564Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:01.5498468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:01.6405482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:01.6405884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:01.9318503Z skip: CUDA is not available. (2.145s)
2022-09-27T15:57:01.9318725Z 
2022-09-27T15:57:01.9319123Z ----------------------------------------------------------------------
2022-09-27T15:57:01.9319432Z Ran 1 test in 2.145s
2022-09-27T15:57:01.9319535Z 
2022-09-27T15:57:01.9319651Z OK (skipped=1)
2022-09-27T15:57:01.9319781Z 
2022-09-27T15:57:01.9319867Z Generating XML reports...
2022-09-27T15:57:01.9364708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155659.xml
2022-09-27T15:57:03.2606022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:03.2606408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:03.2614072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:03.2614855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:03.5985335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:03.5994539Z 
2022-09-27T15:57:03.5994683Z Running tests...
2022-09-27T15:57:03.5995340Z ----------------------------------------------------------------------
2022-09-27T15:57:03.8253328Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.225s)
2022-09-27T15:57:03.8253857Z 
2022-09-27T15:57:03.8254070Z ----------------------------------------------------------------------
2022-09-27T15:57:03.8254326Z Ran 1 test in 0.226s
2022-09-27T15:57:03.8254441Z 
2022-09-27T15:57:03.8254513Z OK (skipped=1)
2022-09-27T15:57:03.8254623Z 
2022-09-27T15:57:03.8254719Z Generating XML reports...
2022-09-27T15:57:03.8276008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155703.xml
2022-09-27T15:57:05.0693174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:05.0693554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:05.0701210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:05.0701562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:05.4086695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:05.4096193Z 
2022-09-27T15:57:05.4096294Z Running tests...
2022-09-27T15:57:05.4097304Z ----------------------------------------------------------------------
2022-09-27T15:57:05.4113534Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:57:05.4114052Z 
2022-09-27T15:57:05.4114394Z ----------------------------------------------------------------------
2022-09-27T15:57:05.4114792Z Ran 1 test in 0.002s
2022-09-27T15:57:05.4114998Z 
2022-09-27T15:57:05.4115124Z OK (skipped=1)
2022-09-27T15:57:05.4115317Z 
2022-09-27T15:57:05.4115469Z Generating XML reports...
2022-09-27T15:57:05.4146138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml
2022-09-27T15:57:06.6019547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:06.6020220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:06.6028227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:06.6028870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:06.9417686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:06.9427965Z 
2022-09-27T15:57:06.9428442Z Running tests...
2022-09-27T15:57:06.9428860Z ----------------------------------------------------------------------
2022-09-27T15:57:06.9444374Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:57:06.9444793Z 
2022-09-27T15:57:06.9445069Z ----------------------------------------------------------------------
2022-09-27T15:57:06.9445304Z Ran 1 test in 0.002s
2022-09-27T15:57:06.9445420Z 
2022-09-27T15:57:06.9445495Z OK (skipped=1)
2022-09-27T15:57:06.9445607Z 
2022-09-27T15:57:06.9445739Z Generating XML reports...
2022-09-27T15:57:06.9475752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155706.xml
2022-09-27T15:57:08.1502951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:08.1503608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:08.1512033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:08.1512392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.4978120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:08.4987440Z 
2022-09-27T15:57:08.4987818Z Running tests...
2022-09-27T15:57:08.4988228Z ----------------------------------------------------------------------
2022-09-27T15:57:08.7339010Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23649
2022-09-27T15:57:08.7365354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23650
2022-09-27T15:57:08.7392876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23651
2022-09-27T15:57:09.8263653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:09.8264158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:09.8572527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:09.8573100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:09.8785474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:09.8786094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:09.9040891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:09.9041242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:09.9106368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:09.9106734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:09.9376545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:09.9376962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:10.2879046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:10.2979719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:10.2980369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:10.2980963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:10.2981503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:10.2982024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:10.3086084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:10.3992676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:10.3993268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:10.6444868Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:57:10.6445123Z 
2022-09-27T15:57:10.6445643Z ----------------------------------------------------------------------
2022-09-27T15:57:10.6445897Z Ran 1 test in 2.146s
2022-09-27T15:57:10.6446001Z 
2022-09-27T15:57:10.6446076Z OK (skipped=1)
2022-09-27T15:57:10.6446252Z 
2022-09-27T15:57:10.6446337Z Generating XML reports...
2022-09-27T15:57:10.6493660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155708.xml
2022-09-27T15:57:11.9883388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:11.9884019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:11.9892360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:12.3290620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:12.3291497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:12.3300916Z 
2022-09-27T15:57:12.3301391Z Running tests...
2022-09-27T15:57:12.3301857Z ----------------------------------------------------------------------
2022-09-27T15:57:12.5647522Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23702
2022-09-27T15:57:12.5674787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23703
2022-09-27T15:57:12.5702737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23704
2022-09-27T15:57:13.5924494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:13.5925146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:13.5946335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:13.5946978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:13.6131157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:13.6131682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:13.6293298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:13.6293685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:13.6317471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:13.6317837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:13.6475434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:13.6475840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:13.9678517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:13.9779495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:13.9780232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:13.9780886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:13.9781447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:13.9782018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:13.9885621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:14.0792995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:14.0793629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:14.3751128Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:57:14.3751301Z 
2022-09-27T15:57:14.3751610Z ----------------------------------------------------------------------
2022-09-27T15:57:14.3752432Z Ran 1 test in 2.045s
2022-09-27T15:57:14.3752644Z 
2022-09-27T15:57:14.3752764Z OK (skipped=1)
2022-09-27T15:57:14.3752924Z 
2022-09-27T15:57:14.3753062Z Generating XML reports...
2022-09-27T15:57:14.3799478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml
2022-09-27T15:57:15.6815428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:15.6815818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:15.6823267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:15.6823614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:16.0190894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:16.0200443Z 
2022-09-27T15:57:16.0200544Z Running tests...
2022-09-27T15:57:16.0201399Z ----------------------------------------------------------------------
2022-09-27T15:57:16.2548851Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23755
2022-09-27T15:57:16.2575797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23756
2022-09-27T15:57:16.2602645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23757
2022-09-27T15:57:17.3692178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:17.3692801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:17.3904112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:17.3904516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:17.4023284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:17.4023642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.4049651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:17.4050276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:17.4251810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:17.4252519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.4382738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:17.4383407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.7818740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:17.7919171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:17.7919630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:17.7920254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:17.7920790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:17.7921486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:17.8027227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:17.8027878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:17.8931555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:18.1652875Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:57:18.1653182Z 
2022-09-27T15:57:18.1653701Z ----------------------------------------------------------------------
2022-09-27T15:57:18.1654116Z Ran 1 test in 2.145s
2022-09-27T15:57:18.1654238Z 
2022-09-27T15:57:18.1654317Z OK (skipped=1)
2022-09-27T15:57:18.1654425Z 
2022-09-27T15:57:18.1654513Z Generating XML reports...
2022-09-27T15:57:18.1699436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155716.xml
2022-09-27T15:57:19.5027870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:19.5028546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:19.5036788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:19.5037432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:19.8572924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:19.8584045Z 
2022-09-27T15:57:19.8584457Z Running tests...
2022-09-27T15:57:19.8584932Z ----------------------------------------------------------------------
2022-09-27T15:57:20.0960744Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23808
2022-09-27T15:57:20.0987940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23809
2022-09-27T15:57:20.1015507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23810
2022-09-27T15:57:21.2015961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.2016338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.2170385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.2170853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.2339252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.2339704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.2359994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.2360545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:21.2535197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.2535634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:21.2695395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.2695759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:21.5985140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:21.5985699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:21.5986377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:21.5987321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:21.5987958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:21.5988480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:21.6090634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:21.6997309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:21.6997871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:22.0068843Z skip: Need at least 2 CUDA devices (2.148s)
2022-09-27T15:57:22.0069122Z 
2022-09-27T15:57:22.0069707Z ----------------------------------------------------------------------
2022-09-27T15:57:22.0070020Z Ran 1 test in 2.148s
2022-09-27T15:57:22.0070135Z 
2022-09-27T15:57:22.0070195Z OK (skipped=1)
2022-09-27T15:57:22.0070306Z 
2022-09-27T15:57:22.0070400Z Generating XML reports...
2022-09-27T15:57:22.0115735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155719.xml
2022-09-27T15:57:23.3447247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:23.3447615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:23.3455576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:23.3455972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:23.6824177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:23.6834332Z 
2022-09-27T15:57:23.6834484Z Running tests...
2022-09-27T15:57:23.6834947Z ----------------------------------------------------------------------
2022-09-27T15:57:23.6864082Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-09-27T15:57:23.6864458Z 
2022-09-27T15:57:23.6864668Z ----------------------------------------------------------------------
2022-09-27T15:57:23.6865006Z Ran 1 test in 0.003s
2022-09-27T15:57:23.6865121Z 
2022-09-27T15:57:23.6865194Z OK (skipped=1)
2022-09-27T15:57:23.6865302Z 
2022-09-27T15:57:23.6865373Z Generating XML reports...
2022-09-27T15:57:23.6894187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155723.xml
2022-09-27T15:57:24.8771181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:24.8771575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:24.8779105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:24.8779461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:25.2188269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:25.2197995Z 
2022-09-27T15:57:25.2198099Z Running tests...
2022-09-27T15:57:25.2198523Z ----------------------------------------------------------------------
2022-09-27T15:57:25.4542407Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23871
2022-09-27T15:57:25.4569226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23872
2022-09-27T15:57:25.4595846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23873
2022-09-27T15:57:26.5682880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.5683582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.6004742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.6005139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:26.6085831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.6086448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.6139420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.6139881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.6425159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.6425546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:26.6483132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.6483496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:26.9645525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:26.9700665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:26.9701077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:26.9701698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:26.9702253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:26.9747271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:26.9808224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:26.9809034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:27.0760414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:27.3646454Z skip: Need at least 2 CUDA devices (2.144s)
2022-09-27T15:57:27.3646662Z 
2022-09-27T15:57:27.3646987Z ----------------------------------------------------------------------
2022-09-27T15:57:27.3647232Z Ran 1 test in 2.145s
2022-09-27T15:57:27.3647347Z 
2022-09-27T15:57:27.3647807Z OK (skipped=1)
2022-09-27T15:57:27.3647964Z 
2022-09-27T15:57:27.3648103Z Generating XML reports...
2022-09-27T15:57:27.3693528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155725.xml
2022-09-27T15:57:28.6624228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:28.6624613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:28.6632213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:28.6632581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:29.0000471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:29.0009888Z 
2022-09-27T15:57:29.0010070Z Running tests...
2022-09-27T15:57:29.0010491Z ----------------------------------------------------------------------
2022-09-27T15:57:29.2414002Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23924
2022-09-27T15:57:29.2439445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23925
2022-09-27T15:57:29.2466635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23926
2022-09-27T15:57:30.3068875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.3069533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.3384930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.3385332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.3507887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.3508559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.3823598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.3824410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.3863815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.3864570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.4179424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.4180080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.7393465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:30.7494950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:30.7496117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:30.7496771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:30.7497295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:30.7497818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:30.7503130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:30.7503828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:30.7506139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:31.0517526Z skip: Need at least 3 CUDA devices (2.050s)
2022-09-27T15:57:31.0517824Z 
2022-09-27T15:57:31.0518164Z ----------------------------------------------------------------------
2022-09-27T15:57:31.0518398Z Ran 1 test in 2.051s
2022-09-27T15:57:31.0518513Z 
2022-09-27T15:57:31.0518602Z OK (skipped=1)
2022-09-27T15:57:31.0518713Z 
2022-09-27T15:57:31.0518798Z Generating XML reports...
2022-09-27T15:57:31.0567024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155728.xml
2022-09-27T15:57:32.3754071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:32.3754448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:32.3762221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:32.3762598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:32.7125780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:32.7135050Z 
2022-09-27T15:57:32.7135188Z Running tests...
2022-09-27T15:57:32.7135673Z ----------------------------------------------------------------------
2022-09-27T15:57:32.7175639Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-09-27T15:57:32.7176023Z 
2022-09-27T15:57:32.7176231Z ----------------------------------------------------------------------
2022-09-27T15:57:32.7176537Z Ran 1 test in 0.004s
2022-09-27T15:57:32.7176652Z 
2022-09-27T15:57:32.7176725Z OK (skipped=1)
2022-09-27T15:57:32.7176832Z 
2022-09-27T15:57:32.7176916Z Generating XML reports...
2022-09-27T15:57:32.7206383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml
2022-09-27T15:57:33.9391162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:33.9391784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:33.9400269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:33.9400936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:34.2823007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:34.2833283Z 
2022-09-27T15:57:34.2833664Z Running tests...
2022-09-27T15:57:34.2834267Z ----------------------------------------------------------------------
2022-09-27T15:57:34.2867498Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-09-27T15:57:34.2867896Z 
2022-09-27T15:57:34.2868278Z ----------------------------------------------------------------------
2022-09-27T15:57:34.2868702Z Ran 1 test in 0.003s
2022-09-27T15:57:34.2868836Z 
2022-09-27T15:57:34.2868913Z OK (skipped=1)
2022-09-27T15:57:34.2869025Z 
2022-09-27T15:57:34.2869112Z Generating XML reports...
2022-09-27T15:57:34.2898020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155734.xml
2022-09-27T15:57:35.5130795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:35.5131183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:35.5138800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:35.5139164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:35.8494924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:35.8504199Z 
2022-09-27T15:57:35.8504298Z Running tests...
2022-09-27T15:57:35.8505168Z ----------------------------------------------------------------------
2022-09-27T15:57:36.0846043Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23997
2022-09-27T15:57:36.0872670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23998
2022-09-27T15:57:36.0899873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23999
2022-09-27T15:57:37.1748957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.1749550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.2066947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.2067640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.2151549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.2152189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.2478210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.2478863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.2585322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.2585982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.2901120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.2901752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.6277934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:37.6378605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:37.6379074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:37.6379713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:37.6380251Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:37.6380764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:37.6485840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:37.6544107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6zh_sqi
2022-09-27T15:57:37.6546022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6zh_sqi/_remote_module_non_scriptable.py
2022-09-27T15:57:37.7389501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:37.7390075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:37.7452639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf01zqb2
2022-09-27T15:57:37.7454452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf01zqb2/_remote_module_non_scriptable.py
2022-09-27T15:57:37.7459756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevavsv0q
2022-09-27T15:57:37.7461745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevavsv0q/_remote_module_non_scriptable.py
2022-09-27T15:57:37.7575365Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7577015Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7578701Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7596529Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:57:37.7597892Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:57:37.7599682Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:57:37.7600521Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:57:37.7601911Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:57:37.7603147Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-09-27T15:57:37.7603695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:57:37.7604266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:57:37.7604769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:57:37.7605930Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7607365Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7608990Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7610871Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7612505Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7614214Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7615830Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7617357Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7618508Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7619629Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7620755Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7621880Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7623001Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7624275Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:37.7625596Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:57:38.0949190Z ok (2.244s)
2022-09-27T15:57:38.0949778Z 
2022-09-27T15:57:38.0950283Z ----------------------------------------------------------------------
2022-09-27T15:57:38.0950782Z Ran 1 test in 2.244s
2022-09-27T15:57:38.0950980Z 
2022-09-27T15:57:38.0951077Z OK
2022-09-27T15:57:38.0951220Z 
2022-09-27T15:57:38.0951303Z Generating XML reports...
2022-09-27T15:57:38.0995388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml
2022-09-27T15:57:39.4065350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:39.4065777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:39.4073190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:39.4073557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:39.7549701Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:39.7559173Z 
2022-09-27T15:57:39.7559320Z Running tests...
2022-09-27T15:57:39.7559748Z ----------------------------------------------------------------------
2022-09-27T15:57:39.7602099Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-09-27T15:57:39.7602446Z 
2022-09-27T15:57:39.7602722Z ----------------------------------------------------------------------
2022-09-27T15:57:39.7602967Z Ran 1 test in 0.004s
2022-09-27T15:57:39.7603080Z 
2022-09-27T15:57:39.7603139Z OK (skipped=1)
2022-09-27T15:57:39.7603245Z 
2022-09-27T15:57:39.7603335Z Generating XML reports...
2022-09-27T15:57:39.7632515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155739.xml
2022-09-27T15:57:40.9730195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:40.9730847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:40.9738852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:40.9739482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:41.3153633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:41.3163500Z 
2022-09-27T15:57:41.3163844Z Running tests...
2022-09-27T15:57:41.3164439Z ----------------------------------------------------------------------
2022-09-27T15:57:41.3192693Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-09-27T15:57:41.3193125Z 
2022-09-27T15:57:41.3193496Z ----------------------------------------------------------------------
2022-09-27T15:57:41.3193890Z Ran 1 test in 0.003s
2022-09-27T15:57:41.3194073Z 
2022-09-27T15:57:41.3194195Z OK (skipped=1)
2022-09-27T15:57:41.3194371Z 
2022-09-27T15:57:41.3194502Z Generating XML reports...
2022-09-27T15:57:41.3225125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155741.xml
2022-09-27T15:57:42.5344001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:42.5344620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:42.5352880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:42.5353533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:42.8727176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:42.8736970Z 
2022-09-27T15:57:42.8737385Z Running tests...
2022-09-27T15:57:42.8737795Z ----------------------------------------------------------------------
2022-09-27T15:57:43.1111977Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24070
2022-09-27T15:57:43.1138053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24071
2022-09-27T15:57:43.1165177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24072
2022-09-27T15:57:44.2375362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:44.2375735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:44.2565057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:44.2565636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:44.2622268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:44.2718525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:44.2719392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:44.2719982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:44.2916387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:44.2917043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:44.2974023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:44.2974706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:44.6139740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:44.6184895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:44.6185369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:44.6186010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:44.6186543Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:44.6241292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:44.6291993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:44.6292510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:44.7253598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:45.0213283Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:57:45.0213552Z 
2022-09-27T15:57:45.0214059Z ----------------------------------------------------------------------
2022-09-27T15:57:45.0214395Z Ran 1 test in 2.147s
2022-09-27T15:57:45.0214510Z 
2022-09-27T15:57:45.0214584Z OK (skipped=1)
2022-09-27T15:57:45.0214690Z 
2022-09-27T15:57:45.0214775Z Generating XML reports...
2022-09-27T15:57:45.0259282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155742.xml
2022-09-27T15:57:46.3369230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:46.3369617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:46.3376746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:46.3377104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:46.6767888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:46.6776982Z 
2022-09-27T15:57:46.6777194Z Running tests...
2022-09-27T15:57:46.6777814Z ----------------------------------------------------------------------
2022-09-27T15:57:46.9139028Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24123
2022-09-27T15:57:46.9166326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24124
2022-09-27T15:57:46.9192552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24125
2022-09-27T15:57:48.0268220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.0268628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.0309193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.0309687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.0614436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.0614994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.0654367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.0654885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.0992003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.0992427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.1309486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.1310164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.4708235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:48.4807607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:48.4808195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:48.4809131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:48.4809964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:48.4810804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:48.4916295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:48.5822893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:48.5921728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:48.9246467Z skip: Need at least 3 CUDA devices (2.247s)
2022-09-27T15:57:48.9246788Z 
2022-09-27T15:57:48.9247290Z ----------------------------------------------------------------------
2022-09-27T15:57:48.9247560Z Ran 1 test in 2.247s
2022-09-27T15:57:48.9247683Z 
2022-09-27T15:57:48.9247757Z OK (skipped=1)
2022-09-27T15:57:48.9247867Z 
2022-09-27T15:57:48.9247958Z Generating XML reports...
2022-09-27T15:57:48.9293998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155746.xml
2022-09-27T15:57:50.2567689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:50.2568353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:50.2575277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:50.2575639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:50.5950208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:50.5959826Z 
2022-09-27T15:57:50.5960441Z Running tests...
2022-09-27T15:57:50.5960852Z ----------------------------------------------------------------------
2022-09-27T15:57:50.8347448Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24176
2022-09-27T15:57:50.8371981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24177
2022-09-27T15:57:50.8400657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24178
2022-09-27T15:57:51.9558418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:51.9559122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:51.9578807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:51.9579468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:51.9580390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:51.9581016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:51.9925256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:51.9925884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:51.9944508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:51.9945203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:51.9946154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:51.9946742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.3484383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:52.3585753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:52.3586259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:52.3587002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:52.3587557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:52.3588083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:52.3594678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:52.3596088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:52.3596599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:52.6450196Z skip: Need at least 3 CUDA devices (2.049s)
2022-09-27T15:57:52.6450423Z 
2022-09-27T15:57:52.6450955Z ----------------------------------------------------------------------
2022-09-27T15:57:52.6451216Z Ran 1 test in 2.049s
2022-09-27T15:57:52.6451319Z 
2022-09-27T15:57:52.6451394Z OK (skipped=1)
2022-09-27T15:57:52.6451568Z 
2022-09-27T15:57:52.6451657Z Generating XML reports...
2022-09-27T15:57:52.6496369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155750.xml
2022-09-27T15:57:53.9304772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:53.9305201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:53.9312946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:53.9313501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:54.2704837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:54.2715054Z 
2022-09-27T15:57:54.2715357Z Running tests...
2022-09-27T15:57:54.2715944Z ----------------------------------------------------------------------
2022-09-27T15:57:54.5081309Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24229
2022-09-27T15:57:54.5107170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24230
2022-09-27T15:57:54.5134205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24231
2022-09-27T15:57:55.5759862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:55.5760515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:55.6072712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:55.6073375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:55.6169751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:55.6170173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:55.6306116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:55.6306767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:55.6507103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:55.6507515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:55.6642456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:55.6642839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:55.9988048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:56.0046506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:56.0047056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:56.0047684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:56.0048218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:56.0089039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:56.0154254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:56.0155941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:56.1100942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:56.4182323Z skip: Need at least 3 CUDA devices (2.146s)
2022-09-27T15:57:56.4182562Z 
2022-09-27T15:57:56.4182897Z ----------------------------------------------------------------------
2022-09-27T15:57:56.4183153Z Ran 1 test in 2.147s
2022-09-27T15:57:56.4183269Z 
2022-09-27T15:57:56.4183342Z OK (skipped=1)
2022-09-27T15:57:56.4183498Z 
2022-09-27T15:57:56.4183595Z Generating XML reports...
2022-09-27T15:57:56.4229225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155754.xml
2022-09-27T15:57:57.7253265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:57.7253651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:57.7261850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:57.7262233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.0633750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:58.0643121Z 
2022-09-27T15:57:58.0643365Z Running tests...
2022-09-27T15:57:58.0643723Z ----------------------------------------------------------------------
2022-09-27T15:57:58.3037206Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24282
2022-09-27T15:57:58.3063584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24283
2022-09-27T15:57:58.3090626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24284
2022-09-27T15:57:59.3780243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:59.3780925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:59.4096973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:59.4097718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:59.4102747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:59.4103306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:59.4179969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:59.4180588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:59.4441666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:59.4442304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:59.4521685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:59.4522389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:59.7717619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:59.7818712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:59.7819330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:57:59.7820282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:59.7821003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:59.7821745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:57:59.7827916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:59.7828744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:59.8832156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:00.1137565Z skip: Need at least 3 CUDA devices (2.049s)
2022-09-27T15:58:00.1137908Z 
2022-09-27T15:58:00.1138408Z ----------------------------------------------------------------------
2022-09-27T15:58:00.1138708Z Ran 1 test in 2.049s
2022-09-27T15:58:00.1138844Z 
2022-09-27T15:58:00.1138921Z OK (skipped=1)
2022-09-27T15:58:00.1139019Z 
2022-09-27T15:58:00.1139551Z Generating XML reports...
2022-09-27T15:58:00.1185645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155758.xml
2022-09-27T15:58:01.4013918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:01.4014299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:01.4022356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:01.4022728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:01.7405686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:01.7415142Z 
2022-09-27T15:58:01.7415409Z Running tests...
2022-09-27T15:58:01.7415983Z ----------------------------------------------------------------------
2022-09-27T15:58:01.9749016Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24335
2022-09-27T15:58:01.9775613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24336
2022-09-27T15:58:01.9802768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24337
2022-09-27T15:58:03.0332959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:03.0333561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:03.0485729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:03.0486279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:03.0601667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:03.0602364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:03.0681906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:03.0682484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:03.0849328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:03.0849905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:03.0949156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:03.0949781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:03.4298451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:03.4381859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:03.4382317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:03.4383125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:03.4383668Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:03.4399185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:03.4488815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:03.4489287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:03.5411969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:03.7850052Z skip: Need at least 3 CUDA devices (2.043s)
2022-09-27T15:58:03.7850420Z 
2022-09-27T15:58:03.7850924Z ----------------------------------------------------------------------
2022-09-27T15:58:03.7851200Z Ran 1 test in 2.043s
2022-09-27T15:58:03.7851317Z 
2022-09-27T15:58:03.7851390Z OK (skipped=1)
2022-09-27T15:58:03.7851486Z 
2022-09-27T15:58:03.7851574Z Generating XML reports...
2022-09-27T15:58:03.7898162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml
2022-09-27T15:58:05.0711729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:05.0712092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:05.0719952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:05.0720313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:05.4118932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:05.4128810Z 
2022-09-27T15:58:05.4129222Z Running tests...
2022-09-27T15:58:05.4129814Z ----------------------------------------------------------------------
2022-09-27T15:58:05.6482834Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24388
2022-09-27T15:58:05.6509756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24389
2022-09-27T15:58:05.6536711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24390
2022-09-27T15:58:06.7763646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:06.7764052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:06.7998827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:06.7999396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:06.8093606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:06.8093963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:06.8141095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:06.8141605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:06.8347064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:06.8347554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:06.8475270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:06.8475751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:07.1808590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:07.1909212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:07.1910233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:07.1911509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:07.1912476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:07.1913443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:07.1916490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:07.1917899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:07.1918461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:07.4586188Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:58:07.4586493Z 
2022-09-27T15:58:07.4586814Z ----------------------------------------------------------------------
2022-09-27T15:58:07.4587081Z Ran 1 test in 2.046s
2022-09-27T15:58:07.4587198Z 
2022-09-27T15:58:07.4587272Z OK (skipped=1)
2022-09-27T15:58:07.4587382Z 
2022-09-27T15:58:07.4587456Z Generating XML reports...
2022-09-27T15:58:07.4631691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155805.xml
2022-09-27T15:58:08.7506612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:08.7507005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:08.7514213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:08.7514596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:09.0871563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:09.0881383Z 
2022-09-27T15:58:09.0881506Z Running tests...
2022-09-27T15:58:09.0881942Z ----------------------------------------------------------------------
2022-09-27T15:58:09.3241905Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24441
2022-09-27T15:58:09.3267914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24442
2022-09-27T15:58:09.3295204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24443
2022-09-27T15:58:10.4093474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:10.4094164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:10.4411719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:10.4412352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:10.4638897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:10.4639306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:10.4890345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:10.4890771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:10.4996267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:10.4997028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:10.5223160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:10.5224229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:10.8430833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:10.8531610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:10.8532644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:10.8533264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:10.8533781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:10.8534421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:10.8538449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:10.8539233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:10.8541315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:11.1340554Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:58:11.1340815Z 
2022-09-27T15:58:11.1341161Z ----------------------------------------------------------------------
2022-09-27T15:58:11.1341415Z Ran 1 test in 2.046s
2022-09-27T15:58:11.1341532Z 
2022-09-27T15:58:11.1341605Z OK (skipped=1)
2022-09-27T15:58:11.1341740Z 
2022-09-27T15:58:11.1341838Z Generating XML reports...
2022-09-27T15:58:11.1387140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml
2022-09-27T15:58:12.4595507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.4596174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.4604320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.4604974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.8001510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:12.8011422Z 
2022-09-27T15:58:12.8011825Z Running tests...
2022-09-27T15:58:12.8012264Z ----------------------------------------------------------------------
2022-09-27T15:58:13.0368560Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24494
2022-09-27T15:58:13.0394192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24495
2022-09-27T15:58:13.0421527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24496
2022-09-27T15:58:14.0799973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:14.0800652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:14.1039809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:14.1040407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:14.1124041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:14.1124649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:14.1366041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:14.1366741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:14.1506345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:14.1506708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:14.1820286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:14.1820661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:14.5172903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:14.5274451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:14.5274968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:14.5275911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:14.5276785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:14.5277314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:14.5282728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:14.5283440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:14.5285945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:14.8475305Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:58:14.8475529Z 
2022-09-27T15:58:14.8475860Z ----------------------------------------------------------------------
2022-09-27T15:58:14.8476118Z Ran 1 test in 2.046s
2022-09-27T15:58:14.8476235Z 
2022-09-27T15:58:14.8476319Z OK (skipped=1)
2022-09-27T15:58:14.8476414Z 
2022-09-27T15:58:14.8476502Z Generating XML reports...
2022-09-27T15:58:14.8524176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155812.xml
2022-09-27T15:58:16.1376022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1376448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1383791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1384407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.4760232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:16.4769654Z 
2022-09-27T15:58:16.4769845Z Running tests...
2022-09-27T15:58:16.4770262Z ----------------------------------------------------------------------
2022-09-27T15:58:16.7134405Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24547
2022-09-27T15:58:16.7160195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24548
2022-09-27T15:58:16.7186382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24549
2022-09-27T15:58:17.7887086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:17.7887661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:17.8080164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:17.8080609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:17.8220437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:17.8221000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:17.8408051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:17.8408419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:17.8725929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:17.8726588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:17.9043838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:17.9044545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:18.2475615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:18.2508786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:18.2509235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:18.2509852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:18.2510390Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:18.2576876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:18.2616790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:18.2617334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:18.3589597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:18.6233565Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T15:58:18.6233918Z 
2022-09-27T15:58:18.6234416Z ----------------------------------------------------------------------
2022-09-27T15:58:18.6234671Z Ran 1 test in 2.146s
2022-09-27T15:58:18.6234785Z 
2022-09-27T15:58:18.6234860Z OK (skipped=1)
2022-09-27T15:58:18.6234971Z 
2022-09-27T15:58:18.6235044Z Generating XML reports...
2022-09-27T15:58:18.6281872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml
2022-09-27T15:58:19.9487016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:19.9487627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:19.9496359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:19.9496770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:20.2866858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:20.2877235Z 
2022-09-27T15:58:20.2877513Z Running tests...
2022-09-27T15:58:20.2878108Z ----------------------------------------------------------------------
2022-09-27T15:58:20.5234308Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24600
2022-09-27T15:58:20.5260149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24601
2022-09-27T15:58:20.5287115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24602
2022-09-27T15:58:21.5603117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:21.5604045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:21.5922297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:21.5922994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:21.5923826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:21.5924504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:21.6114984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:21.6115530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:21.6258859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:21.6259483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:21.6443651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:21.6444302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:21.9711047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:21.9812009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:21.9812488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:21.9813174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:21.9813774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:21.9814345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:21.9917880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:22.0825012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:22.0825582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:22.3334158Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:58:22.3334414Z 
2022-09-27T15:58:22.3334731Z ----------------------------------------------------------------------
2022-09-27T15:58:22.3334980Z Ran 1 test in 2.046s
2022-09-27T15:58:22.3335096Z 
2022-09-27T15:58:22.3335169Z OK (skipped=1)
2022-09-27T15:58:22.3335264Z 
2022-09-27T15:58:22.3335365Z Generating XML reports...
2022-09-27T15:58:22.3380074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155820.xml
2022-09-27T15:58:23.6290060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:23.6290443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:23.6298546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:23.6298921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:23.9812127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:23.9822577Z 
2022-09-27T15:58:23.9822672Z Running tests...
2022-09-27T15:58:23.9823146Z ----------------------------------------------------------------------
2022-09-27T15:58:24.2181775Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24653
2022-09-27T15:58:24.2207304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24654
2022-09-27T15:58:24.2234768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24655
2022-09-27T15:58:25.3120544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:25.3121240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:25.3134891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:25.3135520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:25.3396524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:25.3397183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:25.3466533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:25.3466941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:25.3485500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:25.3485868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:25.3720878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:25.3721355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:25.7037990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:25.7038723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:25.7039413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:25.7040193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:25.7040731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:25.7041246Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:25.7046556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:25.7047455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:25.7048028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:26.0286314Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:58:26.0286586Z 
2022-09-27T15:58:26.0287024Z ----------------------------------------------------------------------
2022-09-27T15:58:26.0287423Z Ran 1 test in 2.046s
2022-09-27T15:58:26.0287603Z 
2022-09-27T15:58:26.0287716Z OK (skipped=1)
2022-09-27T15:58:26.0287880Z 
2022-09-27T15:58:26.0288016Z Generating XML reports...
2022-09-27T15:58:26.0333631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155823.xml
2022-09-27T15:58:27.3125723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:27.3126369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:27.3134760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:27.3135683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:27.6507595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:27.6518363Z 
2022-09-27T15:58:27.6518598Z Running tests...
2022-09-27T15:58:27.6519213Z ----------------------------------------------------------------------
2022-09-27T15:58:27.8855180Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24706
2022-09-27T15:58:27.8882127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24707
2022-09-27T15:58:27.8909355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24708
2022-09-27T15:58:28.9336332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.9337023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.9655618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:28.9656236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:28.9749895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.9750490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.9829295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.9829867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:29.0095833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.0096465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.0173260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.0173968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.3433023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:29.3533746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:29.3534537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:29.3534941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:29.3535455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:29.3535995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:29.3541571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:29.3542468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:29.3543049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:29.6956578Z skip: Need at least 2 CUDA devices (2.043s)
2022-09-27T15:58:29.6956836Z 
2022-09-27T15:58:29.6957181Z ----------------------------------------------------------------------
2022-09-27T15:58:29.6957446Z Ran 1 test in 2.044s
2022-09-27T15:58:29.6957561Z 
2022-09-27T15:58:29.6957633Z OK (skipped=1)
2022-09-27T15:58:29.6957740Z 
2022-09-27T15:58:29.6957825Z Generating XML reports...
2022-09-27T15:58:29.7006413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155827.xml
2022-09-27T15:58:30.9872182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:30.9872575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:30.9880039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:30.9880406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:31.3320345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:31.3330064Z 
2022-09-27T15:58:31.3330209Z Running tests...
2022-09-27T15:58:31.3330618Z ----------------------------------------------------------------------
2022-09-27T15:58:31.5687233Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24759
2022-09-27T15:58:31.5713767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24760
2022-09-27T15:58:31.5741454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24761
2022-09-27T15:58:32.6216876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:32.6217499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:32.6249046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:32.6249654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:32.6408417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:32.6409078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:32.6574573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:32.6575115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:32.6609578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:32.6610109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:32.6775418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:32.6776050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:32.9944814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:33.0046031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:33.0046805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:33.0047480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:33.0048241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:33.0048800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:33.0052905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:33.0053869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:33.0054939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:33.2786950Z skip: Need at least 2 CUDA devices (1.945s)
2022-09-27T15:58:33.2787259Z 
2022-09-27T15:58:33.2788006Z ----------------------------------------------------------------------
2022-09-27T15:58:33.2788559Z Ran 1 test in 1.946s
2022-09-27T15:58:33.2788675Z 
2022-09-27T15:58:33.2788750Z OK (skipped=1)
2022-09-27T15:58:33.2788858Z 
2022-09-27T15:58:33.2789009Z Generating XML reports...
2022-09-27T15:58:33.2835199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155831.xml
2022-09-27T15:58:34.5731037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:34.5731398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:34.5739108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:34.5739590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:34.9134148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:34.9143173Z 
2022-09-27T15:58:34.9143325Z Running tests...
2022-09-27T15:58:34.9144060Z ----------------------------------------------------------------------
2022-09-27T15:58:35.1508666Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24812
2022-09-27T15:58:35.1535074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24813
2022-09-27T15:58:35.1562501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24814
2022-09-27T15:58:36.2764438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:36.2764814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:36.2783701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:36.2784359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:36.2831253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:36.2831702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:36.3123732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:36.3124102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:36.3146125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:36.3146858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:36.3192971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:36.3193602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:36.6682018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:36.6682541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:36.6682971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:36.6683642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:36.6684214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:36.6684783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:36.6789236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:36.6790179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:36.7695307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:37.0613141Z skip: Need at least 2 CUDA devices (2.147s)
2022-09-27T15:58:37.0613416Z 
2022-09-27T15:58:37.0613869Z ----------------------------------------------------------------------
2022-09-27T15:58:37.0614253Z Ran 1 test in 2.147s
2022-09-27T15:58:37.0614432Z 
2022-09-27T15:58:37.0614534Z OK (skipped=1)
2022-09-27T15:58:37.0614696Z 
2022-09-27T15:58:37.0614825Z Generating XML reports...
2022-09-27T15:58:37.0660100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155834.xml
2022-09-27T15:58:38.3640844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.3641511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.3649467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.3650088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.7038748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:38.7047774Z 
2022-09-27T15:58:38.7048171Z Running tests...
2022-09-27T15:58:38.7048607Z ----------------------------------------------------------------------
2022-09-27T15:58:38.9387959Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24865
2022-09-27T15:58:38.9415309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24866
2022-09-27T15:58:38.9442397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24867
2022-09-27T15:58:39.9817924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:39.9818475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:39.9818932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:39.9819267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:40.0041624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:40.0042192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:40.0173157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:40.0173821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:40.0174348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:40.0174683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:40.0380250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:40.0380765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:40.3757171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:40.3855087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:40.3855744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:40.3856366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:40.3857090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:40.3858089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:40.3962437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:40.3963090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:40.4870966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:40.7489553Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:58:40.7489832Z 
2022-09-27T15:58:40.7490204Z ----------------------------------------------------------------------
2022-09-27T15:58:40.7490443Z Ran 1 test in 2.044s
2022-09-27T15:58:40.7490558Z 
2022-09-27T15:58:40.7490632Z OK (skipped=1)
2022-09-27T15:58:40.7490741Z 
2022-09-27T15:58:40.7490828Z Generating XML reports...
2022-09-27T15:58:40.7535558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155838.xml
2022-09-27T15:58:42.0363562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:42.0363919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:42.0371955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:42.0372320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:42.3748013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:42.3757693Z 
2022-09-27T15:58:42.3757853Z Running tests...
2022-09-27T15:58:42.3758263Z ----------------------------------------------------------------------
2022-09-27T15:58:42.6144949Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24918
2022-09-27T15:58:42.6170206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24919
2022-09-27T15:58:42.6197560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24920
2022-09-27T15:58:43.6588991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:43.6589647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:43.6611155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:43.6611775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:43.6809870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:43.6810390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:43.6941075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:43.6941732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:43.6967032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:43.6967647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:43.7147649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:43.7148274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:44.0617219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:44.0718452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:44.0719104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:44.0719955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:44.0720929Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:44.0721441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:44.0727033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:44.0728083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:44.0729233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:44.4247416Z skip: Need at least 2 CUDA devices (2.049s)
2022-09-27T15:58:44.4247683Z 
2022-09-27T15:58:44.4248119Z ----------------------------------------------------------------------
2022-09-27T15:58:44.4248517Z Ran 1 test in 2.049s
2022-09-27T15:58:44.4248702Z 
2022-09-27T15:58:44.4248811Z OK (skipped=1)
2022-09-27T15:58:44.4248977Z 
2022-09-27T15:58:44.4249118Z Generating XML reports...
2022-09-27T15:58:44.4295857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155842.xml
2022-09-27T15:58:45.7225474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:45.7225860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:45.7233884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:45.7234249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:46.0630878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:46.0640163Z 
2022-09-27T15:58:46.0640293Z Running tests...
2022-09-27T15:58:46.0640741Z ----------------------------------------------------------------------
2022-09-27T15:58:46.3011617Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24971
2022-09-27T15:58:46.3039096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24972
2022-09-27T15:58:46.3066115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24973
2022-09-27T15:58:47.3963898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:47.3964279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:47.4110547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:47.4110922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:47.4301581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:47.4302099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:47.4450973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:47.4451496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:47.4966914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:47.4967268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:47.5286625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:47.5287023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:47.8715845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:47.8816790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:47.8817395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:47.8818129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:47.8819028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:47.8819745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:47.8923106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:47.9829442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:47.9830163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:48.3116084Z skip: Need at least 2 CUDA devices (2.247s)
2022-09-27T15:58:48.3116407Z 
2022-09-27T15:58:48.3116925Z ----------------------------------------------------------------------
2022-09-27T15:58:48.3117189Z Ran 1 test in 2.247s
2022-09-27T15:58:48.3117308Z 
2022-09-27T15:58:48.3117370Z OK (skipped=1)
2022-09-27T15:58:48.3117482Z 
2022-09-27T15:58:48.3117571Z Generating XML reports...
2022-09-27T15:58:48.3162168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155846.xml
2022-09-27T15:58:49.6227102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:49.6227592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:49.6235352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:49.6235782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:49.9636663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:49.9645798Z 
2022-09-27T15:58:49.9645891Z Running tests...
2022-09-27T15:58:49.9646339Z ----------------------------------------------------------------------
2022-09-27T15:58:49.9662756Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:58:49.9663249Z 
2022-09-27T15:58:49.9663563Z ----------------------------------------------------------------------
2022-09-27T15:58:49.9663811Z Ran 1 test in 0.002s
2022-09-27T15:58:49.9664065Z 
2022-09-27T15:58:49.9664155Z OK (skipped=1)
2022-09-27T15:58:49.9664253Z 
2022-09-27T15:58:49.9664339Z Generating XML reports...
2022-09-27T15:58:49.9693317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155849.xml
2022-09-27T15:58:51.1564612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:51.1565259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:51.1573449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:51.1574114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:51.4971132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:51.4980374Z 
2022-09-27T15:58:51.4980696Z Running tests...
2022-09-27T15:58:51.4981558Z ----------------------------------------------------------------------
2022-09-27T15:58:51.7367054Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25034
2022-09-27T15:58:51.7395703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25035
2022-09-27T15:58:51.7425084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25036
2022-09-27T15:58:52.8106890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.8107489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.8414950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.8415645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.8618340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.8618939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.8680562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.8681194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.8978831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.8979454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.9038814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.9039499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:53.2451146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:53.2507817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:53.2508548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:53.2509414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:53.2509954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:53.2552292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:53.2615379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:53.2616352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:53.3565340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:53.6473899Z skip: Need at least 2 CUDA devices (2.149s)
2022-09-27T15:58:53.6474189Z 
2022-09-27T15:58:53.6474725Z ----------------------------------------------------------------------
2022-09-27T15:58:53.6475119Z Ran 1 test in 2.149s
2022-09-27T15:58:53.6475225Z 
2022-09-27T15:58:53.6475300Z OK (skipped=1)
2022-09-27T15:58:53.6475418Z 
2022-09-27T15:58:53.6475505Z Generating XML reports...
2022-09-27T15:58:53.6520418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155851.xml
2022-09-27T15:58:54.9724616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:54.9724985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:54.9732528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:54.9733113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:55.3107147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:55.3116777Z 
2022-09-27T15:58:55.3117245Z Running tests...
2022-09-27T15:58:55.3117830Z ----------------------------------------------------------------------
2022-09-27T15:58:55.5457937Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25087
2022-09-27T15:58:55.5484921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25088
2022-09-27T15:58:55.5511943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25089
2022-09-27T15:58:56.6233047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:56.6233707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:56.6238257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:56.6238871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:56.6341594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:56.6342098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:56.6590206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:56.6590961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:56.6596445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:56.6596833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:56.6692375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:56.6692831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:57.0081577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:57.0182959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:58:57.0183467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:57.0184262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:57.0185091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:57.0185965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:58:57.0191822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:57.0192559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:57.0193969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:57.3558523Z skip: Need at least 2 CUDA devices (2.044s)
2022-09-27T15:58:57.3558850Z 
2022-09-27T15:58:57.3559364Z ----------------------------------------------------------------------
2022-09-27T15:58:57.3559624Z Ran 1 test in 2.044s
2022-09-27T15:58:57.3559727Z 
2022-09-27T15:58:57.3559800Z OK (skipped=1)
2022-09-27T15:58:57.3559907Z 
2022-09-27T15:58:57.3559994Z Generating XML reports...
2022-09-27T15:58:57.3606804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155855.xml
2022-09-27T15:58:58.6417791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:58.6418188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:58.6425909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:58.6426409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:58.9919752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:58.9929655Z 
2022-09-27T15:58:58.9929765Z Running tests...
2022-09-27T15:58:58.9930347Z ----------------------------------------------------------------------
2022-09-27T15:58:59.2316808Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25140
2022-09-27T15:58:59.2344190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25141
2022-09-27T15:58:59.2370986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25142
2022-09-27T15:59:00.2905001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:00.2905688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:00.3036287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:00.3036659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:00.3225312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:00.3225814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:00.3244205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:00.3244560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.3388924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:00.3389348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.3559503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:00.3559964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.7052997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:00.7154535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:00.7155327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:00.7155802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:00.7156603Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:00.7157412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:00.7162314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:00.7163478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:00.7164114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:00.7231393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrjxway0
2022-09-27T15:59:00.7232002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57j92475
2022-09-27T15:59:00.7232640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrjxway0/_remote_module_non_scriptable.py
2022-09-27T15:59:00.7233464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1gmezio
2022-09-27T15:59:00.7234130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57j92475/_remote_module_non_scriptable.py
2022-09-27T15:59:00.7234948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1gmezio/_remote_module_non_scriptable.py
2022-09-27T15:59:00.7381415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:00.7382086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:00.7382722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:01.2425728Z ok (2.249s)
2022-09-27T15:59:01.2425968Z 
2022-09-27T15:59:01.2426441Z ----------------------------------------------------------------------
2022-09-27T15:59:01.2426873Z Ran 1 test in 2.250s
2022-09-27T15:59:01.2426989Z 
2022-09-27T15:59:01.2427050Z OK
2022-09-27T15:59:01.2427149Z 
2022-09-27T15:59:01.2427241Z Generating XML reports...
2022-09-27T15:59:01.2472944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155858.xml
2022-09-27T15:59:02.5552594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:02.5552978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:02.5560913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:02.5561276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:02.8939859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:02.8949778Z 
2022-09-27T15:59:02.8949938Z Running tests...
2022-09-27T15:59:02.8950406Z ----------------------------------------------------------------------
2022-09-27T15:59:03.1359000Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25202
2022-09-27T15:59:03.1383804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25203
2022-09-27T15:59:03.1412930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25204
2022-09-27T15:59:04.2409761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:04.2410397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:04.2415052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:04.2415599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:04.2760601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:04.2761282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:04.2767234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:04.2767886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:04.2963651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:04.2964039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:04.3289644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:04.3290016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:04.6658487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:04.6759672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:04.6760356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:04.6761379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:04.6762214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:04.6763027Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:04.6866706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:04.7772534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:04.7773569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:05.0462188Z skip: CUDA is not available. (2.151s)
2022-09-27T15:59:05.0462493Z 
2022-09-27T15:59:05.0463018Z ----------------------------------------------------------------------
2022-09-27T15:59:05.0463305Z Ran 1 test in 2.151s
2022-09-27T15:59:05.0463421Z 
2022-09-27T15:59:05.0463491Z OK (skipped=1)
2022-09-27T15:59:05.0463586Z 
2022-09-27T15:59:05.0463672Z Generating XML reports...
2022-09-27T15:59:05.0507990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155902.xml
2022-09-27T15:59:06.3776831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:06.3777209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:06.3786132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:06.3786552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:06.7154800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:06.7164989Z 
2022-09-27T15:59:06.7165448Z Running tests...
2022-09-27T15:59:06.7165852Z ----------------------------------------------------------------------
2022-09-27T15:59:06.7186375Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:59:06.7186876Z 
2022-09-27T15:59:06.7187189Z ----------------------------------------------------------------------
2022-09-27T15:59:06.7187434Z Ran 1 test in 0.002s
2022-09-27T15:59:06.7187548Z 
2022-09-27T15:59:06.7187621Z OK (skipped=1)
2022-09-27T15:59:06.7187718Z 
2022-09-27T15:59:06.7187802Z Generating XML reports...
2022-09-27T15:59:06.7217319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155906.xml
2022-09-27T15:59:07.9197094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:07.9197486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:07.9204897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:07.9205389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:08.2603370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:08.2612267Z 
2022-09-27T15:59:08.2612619Z Running tests...
2022-09-27T15:59:08.2613213Z ----------------------------------------------------------------------
2022-09-27T15:59:08.2634315Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:59:08.2634606Z 
2022-09-27T15:59:08.2635059Z ----------------------------------------------------------------------
2022-09-27T15:59:08.2635298Z Ran 1 test in 0.002s
2022-09-27T15:59:08.2635416Z 
2022-09-27T15:59:08.2635488Z OK (skipped=1)
2022-09-27T15:59:08.2635597Z 
2022-09-27T15:59:08.2635750Z Generating XML reports...
2022-09-27T15:59:08.2664300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155908.xml
2022-09-27T15:59:09.4744911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:09.4745326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:09.4752910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:09.4753276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:09.8136106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:09.8146359Z 
2022-09-27T15:59:09.8146794Z Running tests...
2022-09-27T15:59:09.8147205Z ----------------------------------------------------------------------
2022-09-27T15:59:09.8163234Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:59:09.8163754Z 
2022-09-27T15:59:09.8164318Z ----------------------------------------------------------------------
2022-09-27T15:59:09.8164593Z Ran 1 test in 0.002s
2022-09-27T15:59:09.8164718Z 
2022-09-27T15:59:09.8164798Z OK (skipped=1)
2022-09-27T15:59:09.8164907Z 
2022-09-27T15:59:09.8164982Z Generating XML reports...
2022-09-27T15:59:09.8194836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155909.xml
2022-09-27T15:59:11.0328967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:11.0329378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:11.0337258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:11.0337621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:11.3755981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:11.3766240Z 
2022-09-27T15:59:11.3766672Z Running tests...
2022-09-27T15:59:11.3767257Z ----------------------------------------------------------------------
2022-09-27T15:59:11.3783386Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T15:59:11.3783829Z 
2022-09-27T15:59:11.3784196Z ----------------------------------------------------------------------
2022-09-27T15:59:11.3784431Z Ran 1 test in 0.002s
2022-09-27T15:59:11.3784545Z 
2022-09-27T15:59:11.3784694Z OK (skipped=1)
2022-09-27T15:59:11.3784812Z 
2022-09-27T15:59:11.3784897Z Generating XML reports...
2022-09-27T15:59:11.3815764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155911.xml
2022-09-27T15:59:12.5945110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:12.5945471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:12.5953486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:12.5954054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:12.9334466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:12.9344209Z 
2022-09-27T15:59:12.9344344Z Running tests...
2022-09-27T15:59:12.9345012Z ----------------------------------------------------------------------
2022-09-27T15:59:12.9368780Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T15:59:12.9369199Z 
2022-09-27T15:59:12.9369501Z ----------------------------------------------------------------------
2022-09-27T15:59:12.9369747Z Ran 1 test in 0.002s
2022-09-27T15:59:12.9369861Z 
2022-09-27T15:59:12.9369934Z OK (skipped=1)
2022-09-27T15:59:12.9370042Z 
2022-09-27T15:59:12.9370114Z Generating XML reports...
2022-09-27T15:59:12.9400659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155912.xml
2022-09-27T15:59:14.1569540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:14.1569915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:14.1578103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:14.1578512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:14.4948669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:14.4957367Z 
2022-09-27T15:59:14.4957496Z Running tests...
2022-09-27T15:59:14.4958335Z ----------------------------------------------------------------------
2022-09-27T15:59:14.7313450Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25305
2022-09-27T15:59:14.7338572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25306
2022-09-27T15:59:14.7365899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25307
2022-09-27T15:59:15.7728805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.7729497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.8049470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.8050081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.8117942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.8118569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.8430869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.8431517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.8615447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.8616064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.8934842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.8935469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:16.2304547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:16.2404583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:16.2405242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:16.2406097Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:16.2406998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:16.2407685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:16.2511867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:16.2512566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:16.3418872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:16.6414936Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:59:16.6415540Z 
2022-09-27T15:59:16.6415994Z ----------------------------------------------------------------------
2022-09-27T15:59:16.6416283Z Ran 1 test in 2.146s
2022-09-27T15:59:16.6416398Z 
2022-09-27T15:59:16.6416472Z OK (skipped=1)
2022-09-27T15:59:16.6416581Z 
2022-09-27T15:59:16.6416666Z Generating XML reports...
2022-09-27T15:59:16.6466730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155914.xml
2022-09-27T15:59:17.9457507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:17.9457899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:17.9465773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:17.9466153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:18.2842744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:18.2852284Z 
2022-09-27T15:59:18.2852421Z Running tests...
2022-09-27T15:59:18.2852998Z ----------------------------------------------------------------------
2022-09-27T15:59:18.5205186Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25358
2022-09-27T15:59:18.5232930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25359
2022-09-27T15:59:18.5266938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25360
2022-09-27T15:59:19.5424369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:19.5425059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:19.5434750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:19.5435356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:19.5649003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:19.5649383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:19.5790529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:19.5790943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:19.5802373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:19.5802762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:19.5992631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:19.5993165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:19.9531116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:19.9531522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:19.9531879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:19.9532678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:19.9533212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:19.9533805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:19.9539504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:19.9540603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:20.0544317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:20.3316042Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:59:20.3316236Z 
2022-09-27T15:59:20.3316626Z ----------------------------------------------------------------------
2022-09-27T15:59:20.3316993Z Ran 1 test in 2.046s
2022-09-27T15:59:20.3317126Z 
2022-09-27T15:59:20.3317201Z OK (skipped=1)
2022-09-27T15:59:20.3317311Z 
2022-09-27T15:59:20.3317399Z Generating XML reports...
2022-09-27T15:59:20.3363266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml
2022-09-27T15:59:21.6496462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.6496872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.6504235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.6504873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.9945102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:21.9954384Z 
2022-09-27T15:59:21.9954526Z Running tests...
2022-09-27T15:59:21.9955159Z ----------------------------------------------------------------------
2022-09-27T15:59:21.9971483Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T15:59:21.9972010Z 
2022-09-27T15:59:21.9972264Z ----------------------------------------------------------------------
2022-09-27T15:59:21.9972509Z Ran 1 test in 0.002s
2022-09-27T15:59:21.9972626Z 
2022-09-27T15:59:21.9972699Z OK (skipped=1)
2022-09-27T15:59:21.9972807Z 
2022-09-27T15:59:21.9972878Z Generating XML reports...
2022-09-27T15:59:22.0002349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155921.xml
2022-09-27T15:59:23.1914804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:23.1915447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:23.1924188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:23.1924580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:23.5305236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:23.5314770Z 
2022-09-27T15:59:23.5314888Z Running tests...
2022-09-27T15:59:23.5315280Z ----------------------------------------------------------------------
2022-09-27T15:59:23.5333116Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:59:23.5333509Z 
2022-09-27T15:59:23.5333868Z ----------------------------------------------------------------------
2022-09-27T15:59:23.5334301Z Ran 1 test in 0.002s
2022-09-27T15:59:23.5334505Z 
2022-09-27T15:59:23.5334609Z OK (skipped=1)
2022-09-27T15:59:23.5334720Z 
2022-09-27T15:59:23.5334807Z Generating XML reports...
2022-09-27T15:59:23.5364098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155923.xml
2022-09-27T15:59:24.7195043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:24.7195698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:24.7202627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:24.7203032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:25.0582804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:25.0592986Z 
2022-09-27T15:59:25.0593459Z Running tests...
2022-09-27T15:59:25.0593866Z ----------------------------------------------------------------------
2022-09-27T15:59:25.2939639Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25431
2022-09-27T15:59:25.2966181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25432
2022-09-27T15:59:25.2993649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25433
2022-09-27T15:59:26.4299813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:26.4310751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:26.4365669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:26.4366159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:26.4431784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:26.4432283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:26.4652303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:26.4652893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:26.4724004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:26.4724673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:26.4786895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:26.4787502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:26.8050583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:26.8150875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:26.8151495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:26.8152436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:26.8153308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:26.8154200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:26.8257349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:26.9163966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:26.9164675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:27.2045226Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T15:59:27.2045442Z 
2022-09-27T15:59:27.2046082Z ----------------------------------------------------------------------
2022-09-27T15:59:27.2046370Z Ran 1 test in 2.145s
2022-09-27T15:59:27.2046517Z 
2022-09-27T15:59:27.2046658Z OK (skipped=1)
2022-09-27T15:59:27.2046765Z 
2022-09-27T15:59:27.2046851Z Generating XML reports...
2022-09-27T15:59:27.2091640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155925.xml
2022-09-27T15:59:28.5272519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:28.5273160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:28.5281090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:28.5281753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:28.8686974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:28.8696740Z 
2022-09-27T15:59:28.8697015Z Running tests...
2022-09-27T15:59:28.8697641Z ----------------------------------------------------------------------
2022-09-27T15:59:29.1087673Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25484
2022-09-27T15:59:29.1113871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25485
2022-09-27T15:59:29.1140847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25486
2022-09-27T15:59:30.1787658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.1788043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.2036865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.2037517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.2108133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.2108811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.2371285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.2371938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.2474044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.2474668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.2794567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.2795249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.6043329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:30.6043798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:30.6044179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:30.6045173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:30.6045841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:30.6046353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:30.6147943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:30.7056331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:30.7056956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:31.0192363Z skip: Need at least 2 CUDA devices (2.149s)
2022-09-27T15:59:31.0193592Z 
2022-09-27T15:59:31.0193957Z ----------------------------------------------------------------------
2022-09-27T15:59:31.0194328Z Ran 1 test in 2.149s
2022-09-27T15:59:31.0194523Z 
2022-09-27T15:59:31.0194663Z OK (skipped=1)
2022-09-27T15:59:31.0194848Z 
2022-09-27T15:59:31.0195009Z Generating XML reports...
2022-09-27T15:59:31.0238881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155928.xml
2022-09-27T15:59:32.3614376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:32.3614728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:32.3622567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:32.3623085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:32.7009837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:32.7020374Z 
2022-09-27T15:59:32.7020734Z Running tests...
2022-09-27T15:59:32.7021347Z ----------------------------------------------------------------------
2022-09-27T15:59:32.7045321Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-09-27T15:59:32.7045781Z 
2022-09-27T15:59:32.7046128Z ----------------------------------------------------------------------
2022-09-27T15:59:32.7046532Z Ran 1 test in 0.003s
2022-09-27T15:59:32.7046716Z 
2022-09-27T15:59:32.7046820Z OK (skipped=1)
2022-09-27T15:59:32.7047002Z 
2022-09-27T15:59:32.7047163Z Generating XML reports...
2022-09-27T15:59:32.7077513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155932.xml
2022-09-27T15:59:33.8958955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:33.8959571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:33.8967380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:33.8968031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:34.2351588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:34.2361605Z 
2022-09-27T15:59:34.2362022Z Running tests...
2022-09-27T15:59:34.2362415Z ----------------------------------------------------------------------
2022-09-27T15:59:34.4732790Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25547
2022-09-27T15:59:34.4759058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25548
2022-09-27T15:59:34.4785966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25549
2022-09-27T15:59:35.5324861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:35.5325486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:35.5330651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:35.5331213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:35.5380506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:35.5381434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:35.5684167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:35.5685225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:35.5686604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:35.5687307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:35.5736578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:35.5737210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:35.9080063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:35.9180677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:35.9181213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:35.9181878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:35.9182417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:35.9182944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:35.9286716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:36.0193251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:36.0193663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:36.2833842Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T15:59:36.2834151Z 
2022-09-27T15:59:36.2834575Z ----------------------------------------------------------------------
2022-09-27T15:59:36.2834849Z Ran 1 test in 2.047s
2022-09-27T15:59:36.2834968Z 
2022-09-27T15:59:36.2835028Z OK (skipped=1)
2022-09-27T15:59:36.2835139Z 
2022-09-27T15:59:36.2835226Z Generating XML reports...
2022-09-27T15:59:36.2879424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155934.xml
2022-09-27T15:59:37.5634862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:37.5635341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:37.5642547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:37.5642997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:37.9000505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:37.9009774Z 
2022-09-27T15:59:37.9010162Z Running tests...
2022-09-27T15:59:37.9010633Z ----------------------------------------------------------------------
2022-09-27T15:59:38.1361609Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25600
2022-09-27T15:59:38.1387695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25601
2022-09-27T15:59:38.1416545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25602
2022-09-27T15:59:39.2305467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:39.2306155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:39.2320635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:39.2321226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:39.2661063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:39.2662040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:39.2672821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:39.2673474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:39.2822802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:39.2823432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:39.3143801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:39.3144664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:39.6337860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:39.6439266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:39.6439988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:39.6441085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:39.6441680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:39.6442209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:39.6447306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:39.6448664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:39.6449078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:39.9465562Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T15:59:39.9465878Z 
2022-09-27T15:59:39.9466397Z ----------------------------------------------------------------------
2022-09-27T15:59:39.9466745Z Ran 1 test in 2.045s
2022-09-27T15:59:39.9466859Z 
2022-09-27T15:59:39.9466934Z OK (skipped=1)
2022-09-27T15:59:39.9467041Z 
2022-09-27T15:59:39.9467114Z Generating XML reports...
2022-09-27T15:59:39.9511263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155937.xml
2022-09-27T15:59:41.2368270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:41.2368660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:41.2375768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:41.2376147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:41.5834188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:41.5843189Z 
2022-09-27T15:59:41.5843272Z Running tests...
2022-09-27T15:59:41.5843696Z ----------------------------------------------------------------------
2022-09-27T15:59:41.8202253Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25653
2022-09-27T15:59:41.8229338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25654
2022-09-27T15:59:41.8256197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25655
2022-09-27T15:59:42.8892880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.8893281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.9133200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.9133581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.9217134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.9217595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:42.9274316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.9487669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.9488493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.9488856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:42.9617001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.9617622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:43.2890423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:43.2992204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:43.2993013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:43.2993490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:43.2994081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:43.2994623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:43.2999909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:43.3000463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:43.3002740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:43.6306126Z skip: Need at least 2 CUDA devices (2.046s)
2022-09-27T15:59:43.6306318Z 
2022-09-27T15:59:43.6306684Z ----------------------------------------------------------------------
2022-09-27T15:59:43.6306925Z Ran 1 test in 2.046s
2022-09-27T15:59:43.6307041Z 
2022-09-27T15:59:43.6307118Z OK (skipped=1)
2022-09-27T15:59:43.6307244Z 
2022-09-27T15:59:43.6307377Z Generating XML reports...
2022-09-27T15:59:43.6351801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155941.xml
2022-09-27T15:59:44.9126221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:44.9126627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:44.9134077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:44.9134592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:45.2534883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:45.2543835Z 
2022-09-27T15:59:45.2544250Z Running tests...
2022-09-27T15:59:45.2544949Z ----------------------------------------------------------------------
2022-09-27T15:59:45.4932531Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25706
2022-09-27T15:59:45.4958754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25707
2022-09-27T15:59:45.4985957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25708
2022-09-27T15:59:46.5925086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5925760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.5973837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5974491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.6273623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.6274410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.6318585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.6319294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.6325507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.6326090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.6634281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.6634918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.9802886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:46.9904186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:46.9904855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:46.9905815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:46.9906463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:46.9906981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:46.9912065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:46.9913095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:46.9913515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:47.3036287Z skip: Need at least 2 CUDA devices (2.049s)
2022-09-27T15:59:47.3036538Z 
2022-09-27T15:59:47.3036934Z ----------------------------------------------------------------------
2022-09-27T15:59:47.3037196Z Ran 1 test in 2.049s
2022-09-27T15:59:47.3037313Z 
2022-09-27T15:59:47.3037398Z OK (skipped=1)
2022-09-27T15:59:47.3037508Z 
2022-09-27T15:59:47.3037598Z Generating XML reports...
2022-09-27T15:59:47.3082049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155945.xml
2022-09-27T15:59:48.6149207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:48.6149599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:48.6157241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:48.6157765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:48.9543268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:48.9552403Z 
2022-09-27T15:59:48.9552560Z Running tests...
2022-09-27T15:59:48.9553208Z ----------------------------------------------------------------------
2022-09-27T15:59:49.1928563Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25759
2022-09-27T15:59:49.1953931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25760
2022-09-27T15:59:49.1980832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25761
2022-09-27T15:59:50.2877454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:50.2878088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:50.3097081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:50.3097848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:50.3205891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:50.3206458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:50.3282388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:50.3282787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:50.3447974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:50.3448345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:50.3612596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:50.3612991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:50.6875573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:50.6975895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:50.6976319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:50.6976941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:50.6977465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:50.6977987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:50.6983814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:50.6984695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:50.6985463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:51.0027498Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T15:59:51.0027797Z 
2022-09-27T15:59:51.0028294Z ----------------------------------------------------------------------
2022-09-27T15:59:51.0028767Z Ran 1 test in 2.047s
2022-09-27T15:59:51.0028958Z 
2022-09-27T15:59:51.0029086Z OK (skipped=1)
2022-09-27T15:59:51.0029269Z 
2022-09-27T15:59:51.0029397Z Generating XML reports...
2022-09-27T15:59:51.0072547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155948.xml
2022-09-27T15:59:52.3098331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:52.3099318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:52.3107766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:52.3108595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:52.6507900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:52.6517811Z 
2022-09-27T15:59:52.6518127Z Running tests...
2022-09-27T15:59:52.6518722Z ----------------------------------------------------------------------
2022-09-27T15:59:52.8842125Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.232s)
2022-09-27T15:59:52.8843028Z 
2022-09-27T15:59:52.8843393Z ----------------------------------------------------------------------
2022-09-27T15:59:52.8843787Z Ran 1 test in 0.232s
2022-09-27T15:59:52.8843978Z 
2022-09-27T15:59:52.8844100Z OK (skipped=1)
2022-09-27T15:59:52.8844289Z 
2022-09-27T15:59:52.8844432Z Generating XML reports...
2022-09-27T15:59:52.8866469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155952.xml
2022-09-27T15:59:54.1721355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:54.1721728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:54.1729305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:54.1729654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:54.5111723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:54.5120970Z 
2022-09-27T15:59:54.5121052Z Running tests...
2022-09-27T15:59:54.5121811Z ----------------------------------------------------------------------
2022-09-27T15:59:54.7417107Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.229s)
2022-09-27T15:59:54.7417621Z 
2022-09-27T15:59:54.7417842Z ----------------------------------------------------------------------
2022-09-27T15:59:54.7418078Z Ran 1 test in 0.229s
2022-09-27T15:59:54.7418195Z 
2022-09-27T15:59:54.7418268Z OK (skipped=1)
2022-09-27T15:59:54.7418375Z 
2022-09-27T15:59:54.7418463Z Generating XML reports...
2022-09-27T15:59:54.7440370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155954.xml
2022-09-27T15:59:56.0090337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:56.0091023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:56.0099035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:56.0099655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:56.3475392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:56.3485348Z 
2022-09-27T15:59:56.3485830Z Running tests...
2022-09-27T15:59:56.3486222Z ----------------------------------------------------------------------
2022-09-27T15:59:56.3514407Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-09-27T15:59:56.3514691Z 
2022-09-27T15:59:56.3514920Z ----------------------------------------------------------------------
2022-09-27T15:59:56.3515227Z Ran 1 test in 0.003s
2022-09-27T15:59:56.3515343Z 
2022-09-27T15:59:56.3515419Z OK (skipped=1)
2022-09-27T15:59:56.3515530Z 
2022-09-27T15:59:56.3515617Z Generating XML reports...
2022-09-27T15:59:56.3544246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155956.xml
2022-09-27T15:59:57.5611904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:57.5612280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:57.5619460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:57.5619872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:57.8983123Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:57.8992436Z 
2022-09-27T15:59:57.8992563Z Running tests...
2022-09-27T15:59:57.8993138Z ----------------------------------------------------------------------
2022-09-27T15:59:58.1341562Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25842
2022-09-27T15:59:58.1368572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25843
2022-09-27T15:59:58.1395068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25844
2022-09-27T15:59:59.2012932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:59.2013577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:59.2061843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:59.2062441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:59.2346483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:59.2347139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:59.2406794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:59.2407426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:59.2569103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:59.2569763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:59.2902024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:59.2902651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:59.6289455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:59.6390472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:59.6390922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:59:59.6391536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:59.6392072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:59.6392602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T15:59:59.6398178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:59.6399018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:59.6400492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:59.6506891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:59:59.6608038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:59:59.6608723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T15:59:59.6609740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:59:59.6610677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T15:59:59.6611215Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:00.1446520Z ok (2.245s)
2022-09-27T16:00:00.1446727Z 
2022-09-27T16:00:00.1447258Z ----------------------------------------------------------------------
2022-09-27T16:00:00.1447695Z Ran 1 test in 2.245s
2022-09-27T16:00:00.1447838Z 
2022-09-27T16:00:00.1447890Z OK
2022-09-27T16:00:00.1447983Z 
2022-09-27T16:00:00.1448080Z Generating XML reports...
2022-09-27T16:00:00.1493385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155957.xml
2022-09-27T16:00:01.4504669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:01.4505053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:01.4512894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:01.4513337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:01.7920902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:01.7931284Z 
2022-09-27T16:00:01.7931816Z Running tests...
2022-09-27T16:00:01.7932226Z ----------------------------------------------------------------------
2022-09-27T16:00:02.0343467Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25904
2022-09-27T16:00:02.0370554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25905
2022-09-27T16:00:02.0397336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25906
2022-09-27T16:00:03.1042814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.1043469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.1308154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.1308787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.1363672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.1364321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.1499018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.1499621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.1654350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.1655026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.1834638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.1835247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.5151774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:03.5152453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:03.5152875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:03.5153512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:03.5154038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:03.5154570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:03.5257811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:03.6162163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:03.6162607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:03.6163638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:03.6268317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:03.6269055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:03.6269889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:03.6270443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:03.6368308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:04.0449385Z ok (2.251s)
2022-09-27T16:00:04.0449610Z 
2022-09-27T16:00:04.0449939Z ----------------------------------------------------------------------
2022-09-27T16:00:04.0450195Z Ran 1 test in 2.252s
2022-09-27T16:00:04.0450300Z 
2022-09-27T16:00:04.0450364Z OK
2022-09-27T16:00:04.0450461Z 
2022-09-27T16:00:04.0450557Z Generating XML reports...
2022-09-27T16:00:04.0495761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160001.xml
2022-09-27T16:00:05.3481071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:05.3481445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:05.3489194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:05.3489701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:05.6873453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:05.6882704Z 
2022-09-27T16:00:05.6882838Z Running tests...
2022-09-27T16:00:05.6883415Z ----------------------------------------------------------------------
2022-09-27T16:00:05.9240734Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25963
2022-09-27T16:00:05.9267884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25964
2022-09-27T16:00:05.9294676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25965
2022-09-27T16:00:06.9935501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:06.9936137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:07.0248335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:07.0248970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:07.0273075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:07.0273432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:07.0533592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:07.0534181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:07.0597543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:07.0598204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:07.0982593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:07.0983264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:07.4407086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:07.4407513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:07.4407975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:07.4408927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:07.4409602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:07.4410173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:07.4414947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:07.4415502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:07.4416279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:07.7341962Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T16:00:07.7342305Z 
2022-09-27T16:00:07.7342808Z ----------------------------------------------------------------------
2022-09-27T16:00:07.7343069Z Ran 1 test in 2.046s
2022-09-27T16:00:07.7343186Z 
2022-09-27T16:00:07.7343261Z OK (skipped=1)
2022-09-27T16:00:07.7343356Z 
2022-09-27T16:00:07.7343444Z Generating XML reports...
2022-09-27T16:00:07.7388924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160005.xml
2022-09-27T16:00:09.0195379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:09.0195790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:09.0202983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:09.0203567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:09.3700048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:09.3709962Z 
2022-09-27T16:00:09.3710106Z Running tests...
2022-09-27T16:00:09.3710498Z ----------------------------------------------------------------------
2022-09-27T16:00:09.3729701Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T16:00:09.3730207Z 
2022-09-27T16:00:09.3730670Z ----------------------------------------------------------------------
2022-09-27T16:00:09.3730921Z Ran 1 test in 0.002s
2022-09-27T16:00:09.3731038Z 
2022-09-27T16:00:09.3731179Z OK (skipped=1)
2022-09-27T16:00:09.3731276Z 
2022-09-27T16:00:09.3731363Z Generating XML reports...
2022-09-27T16:00:09.3761367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160009.xml
2022-09-27T16:00:10.5618148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:10.5618821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:10.5627860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:10.5628517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:10.9015748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:10.9025867Z 
2022-09-27T16:00:10.9026185Z Running tests...
2022-09-27T16:00:10.9026788Z ----------------------------------------------------------------------
2022-09-27T16:00:11.1419650Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26026
2022-09-27T16:00:11.1446525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26027
2022-09-27T16:00:11.1474639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26028
2022-09-27T16:00:12.2201882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:12.2202489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:12.2286231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:12.2286867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:12.2345409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:12.2345944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:12.2572179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:12.2572775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:12.2665005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:12.2665666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:12.2708838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:12.2709492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:12.6054203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:12.6054917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:12.6055622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:12.6056589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:12.6057126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:12.6057648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:12.6063076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:12.6063816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:12.6066284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:12.9521387Z ok (2.049s)
2022-09-27T16:00:12.9521631Z 
2022-09-27T16:00:12.9522083Z ----------------------------------------------------------------------
2022-09-27T16:00:12.9522475Z Ran 1 test in 2.049s
2022-09-27T16:00:12.9522653Z 
2022-09-27T16:00:12.9522745Z OK
2022-09-27T16:00:12.9522896Z 
2022-09-27T16:00:12.9523047Z Generating XML reports...
2022-09-27T16:00:12.9568760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160010.xml
2022-09-27T16:00:14.2424441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:14.2425181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:14.2433581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:14.2434199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:14.5936878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:14.5946799Z 
2022-09-27T16:00:14.5947106Z Running tests...
2022-09-27T16:00:14.5947734Z ----------------------------------------------------------------------
2022-09-27T16:00:14.8293554Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26079
2022-09-27T16:00:14.8320129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26080
2022-09-27T16:00:14.8347123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26081
2022-09-27T16:00:15.9676321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:15.9677003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:15.9793297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:15.9793823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:15.9917398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:15.9917884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:16.0022814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:16.0023462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:16.0150191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:16.0150849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:16.0262307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:16.0262964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:16.3617088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:16.3718451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:16.3719053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:16.3719730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:16.3720426Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:16.3721486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:16.3726539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:16.3727371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:16.3728253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:16.3730864Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3731435Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3733568Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3737879Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3738501Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3739005Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3742691Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3743149Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3747476Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3751683Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3752198Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3755464Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3756583Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3758943Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3762087Z STAGE:2022-09-27 16:00:16 26079:26079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3768504Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:16.3771965Z STAGE:2022-09-27 16:00:16 26080:26080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.3772434Z STAGE:2022-09-27 16:00:16 26081:26081 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:16.8396467Z ok (2.245s)
2022-09-27T16:00:16.8396704Z 
2022-09-27T16:00:16.8397068Z ----------------------------------------------------------------------
2022-09-27T16:00:16.8397324Z Ran 1 test in 2.245s
2022-09-27T16:00:16.8397487Z 
2022-09-27T16:00:16.8397556Z OK
2022-09-27T16:00:16.8397661Z 
2022-09-27T16:00:16.8397758Z Generating XML reports...
2022-09-27T16:00:16.8442884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160014.xml
2022-09-27T16:00:18.1427810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:18.1428548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:18.1437362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:18.1438547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:18.4838655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:18.4848858Z 
2022-09-27T16:00:18.4849276Z Running tests...
2022-09-27T16:00:18.4849705Z ----------------------------------------------------------------------
2022-09-27T16:00:18.7213191Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26135
2022-09-27T16:00:18.7240041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26136
2022-09-27T16:00:18.7266938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26137
2022-09-27T16:00:19.7932489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.7932882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.8011128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.8011632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.8267457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.8267877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:19.8345997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.8346478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.8355346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.8355866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:19.8663695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.8664355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:20.2096003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:20.2096869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:20.2097714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:20.2098817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:20.2099712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:20.2100563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:20.2201942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:20.3106197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:20.3106576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:20.6317420Z ok (2.146s)
2022-09-27T16:00:20.6317649Z 
2022-09-27T16:00:20.6318132Z ----------------------------------------------------------------------
2022-09-27T16:00:20.6318617Z Ran 1 test in 2.147s
2022-09-27T16:00:20.6318769Z 
2022-09-27T16:00:20.6318844Z OK
2022-09-27T16:00:20.6318934Z 
2022-09-27T16:00:20.6319029Z Generating XML reports...
2022-09-27T16:00:20.6364566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160018.xml
2022-09-27T16:00:21.9619156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:21.9619534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:21.9627684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:21.9628039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:22.3021963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:22.3031795Z 
2022-09-27T16:00:22.3032293Z Running tests...
2022-09-27T16:00:22.3032944Z ----------------------------------------------------------------------
2022-09-27T16:00:22.3048671Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-09-27T16:00:22.3049373Z 
2022-09-27T16:00:22.3049647Z ----------------------------------------------------------------------
2022-09-27T16:00:22.3050004Z Ran 1 test in 0.002s
2022-09-27T16:00:22.3050127Z 
2022-09-27T16:00:22.3050201Z OK (skipped=1)
2022-09-27T16:00:22.3050331Z 
2022-09-27T16:00:22.3050444Z Generating XML reports...
2022-09-27T16:00:22.3079729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml
2022-09-27T16:00:23.5105338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:23.5105786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:23.5113115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:23.5113487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:23.8514599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:23.8524524Z 
2022-09-27T16:00:23.8525004Z Running tests...
2022-09-27T16:00:23.8525402Z ----------------------------------------------------------------------
2022-09-27T16:00:24.0894600Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26198
2022-09-27T16:00:24.0921225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26199
2022-09-27T16:00:24.0948786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26200
2022-09-27T16:00:25.1611599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:25.1612255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:25.1926602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:25.1927180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:25.1939978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:25.1940367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:25.2147345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:25.2147842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:25.2268550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:25.2268983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:25.2477416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:25.2477777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:25.5739408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:25.5740067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:25.5740962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:25.5741668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:25.5742376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:25.5742897Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:25.5844177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:25.6747916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:25.6749979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:25.6955916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:25.6956871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:25.6957846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:25.6958540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:25.6959450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:25.6960460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:25.6961459Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6961899Z 
2022-09-27T16:00:25.6962309Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6967269Z STAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6968318Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6971784Z STAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6974922Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6978693Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6982689Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6984408Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6986215Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6989703Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6990521Z STAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.6992439Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.6996393Z STAGE:2022-09-27 16:00:25 26198:26198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.7002449Z STAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:25.7005804Z STAGE:2022-09-27 16:00:25 26199:26199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:25.7006675Z STAGE:2022-09-27 16:00:25 26200:26200 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:26.0999575Z ok (2.247s)
2022-09-27T16:00:26.0999823Z 
2022-09-27T16:00:26.1000248Z ----------------------------------------------------------------------
2022-09-27T16:00:26.1000626Z Ran 1 test in 2.247s
2022-09-27T16:00:26.1000807Z 
2022-09-27T16:00:26.1000897Z OK
2022-09-27T16:00:26.1001275Z 
2022-09-27T16:00:26.1001424Z Generating XML reports...
2022-09-27T16:00:26.1046716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160023.xml
2022-09-27T16:00:27.4050631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:27.4051273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:27.4059740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:27.4060383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:27.7432118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:27.7441529Z 
2022-09-27T16:00:27.7441967Z Running tests...
2022-09-27T16:00:27.7442550Z ----------------------------------------------------------------------
2022-09-27T16:00:27.9831339Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26263
2022-09-27T16:00:27.9856678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26264
2022-09-27T16:00:27.9883519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26265
2022-09-27T16:00:29.0932415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:29.0933066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:29.1023783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:29.1024313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:29.1206614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:29.1207258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:29.1276486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:29.1276963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:29.1378434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:29.1378910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:29.1539828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:29.1540463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:29.4727775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:29.4828776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:29.4829482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:29.4830561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:29.4831394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:29.4831918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:29.4837219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:29.4839290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:29.4839909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:29.4840785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:29.4945347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:29.4945928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:29.4946523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:29.4947063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:29.4949184Z STAGE:2022-09-27 16:00:29 26264:26264 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:29.5044205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:29.5049901Z STAGE:2022-09-27 16:00:29 26265:26265 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:29.5054474Z STAGE:2022-09-27 16:00:29 26265:26265 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:29.5055384Z STAGE:2022-09-27 16:00:29 26264:26264 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:29.5058929Z STAGE:2022-09-27 16:00:29 26265:26265 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:29.5066511Z STAGE:2022-09-27 16:00:29 26264:26264 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:00:29.5070237Z STAGE:2022-09-27 16:00:29 26264:26264 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:29.5071305Z STAGE:2022-09-27 16:00:29 26265:26265 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:00:29.9933526Z ok (2.249s)
2022-09-27T16:00:29.9933820Z 
2022-09-27T16:00:29.9934352Z ----------------------------------------------------------------------
2022-09-27T16:00:29.9934684Z Ran 1 test in 2.249s
2022-09-27T16:00:29.9934804Z 
2022-09-27T16:00:29.9934867Z OK
2022-09-27T16:00:29.9934946Z 
2022-09-27T16:00:29.9935045Z Generating XML reports...
2022-09-27T16:00:29.9979392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160027.xml
2022-09-27T16:00:31.2955442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:31.2955829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:31.2963390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:31.2963772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:31.6353385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:31.6363021Z 
2022-09-27T16:00:31.6363291Z Running tests...
2022-09-27T16:00:31.6363926Z ----------------------------------------------------------------------
2022-09-27T16:00:31.8710049Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26324
2022-09-27T16:00:31.8739670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26325
2022-09-27T16:00:31.8767016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26326
2022-09-27T16:00:32.9876993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:32.9877580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:32.9981875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:32.9982570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:33.0037113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:33.0037780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:33.0237998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:33.0238575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:33.0349015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:33.0349663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:33.0397966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:33.0398347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:33.3828517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:33.3829248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:33.3829697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:33.3830319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:33.3830853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:33.3831382Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:33.3835699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:33.3836306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:33.3837266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:33.6817179Z ok (2.045s)
2022-09-27T16:00:33.6817424Z 
2022-09-27T16:00:33.6817918Z ----------------------------------------------------------------------
2022-09-27T16:00:33.6818184Z Ran 1 test in 2.045s
2022-09-27T16:00:33.6818301Z 
2022-09-27T16:00:33.6818348Z OK
2022-09-27T16:00:33.6818443Z 
2022-09-27T16:00:33.6818536Z Generating XML reports...
2022-09-27T16:00:33.6864248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160031.xml
2022-09-27T16:00:34.9754252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:34.9754923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:34.9763233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:34.9763898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:35.3176072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:35.3185745Z 
2022-09-27T16:00:35.3186183Z Running tests...
2022-09-27T16:00:35.3186808Z ----------------------------------------------------------------------
2022-09-27T16:00:35.5457095Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.227s)
2022-09-27T16:00:35.5458018Z 
2022-09-27T16:00:35.5458393Z ----------------------------------------------------------------------
2022-09-27T16:00:35.5458797Z Ran 1 test in 0.227s
2022-09-27T16:00:35.5458990Z 
2022-09-27T16:00:35.5459106Z OK (skipped=1)
2022-09-27T16:00:35.5459575Z 
2022-09-27T16:00:35.5459712Z Generating XML reports...
2022-09-27T16:00:35.5481411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160035.xml
2022-09-27T16:00:36.7915424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:36.7915818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:36.7923661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:36.7924036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:37.1308459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:37.1317006Z 
2022-09-27T16:00:37.1317238Z Running tests...
2022-09-27T16:00:37.1317671Z ----------------------------------------------------------------------
2022-09-27T16:00:37.3668810Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26387
2022-09-27T16:00:37.3694348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26388
2022-09-27T16:00:37.3721617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26389
2022-09-27T16:00:38.4631558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:38.4631980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:38.4698067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:38.4698578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:38.4969087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:38.4969665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:38.5044347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:38.5044878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:38.5136843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:38.5137477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:38.5453651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:38.5454313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:38.8918089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:38.8918725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:38.8919289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:38.8920201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:38.8920755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:38.8921296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:38.8926596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:38.8927049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:38.8928810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:38.8929319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:38.9133064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:38.9133621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:38.9134200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:38.9134733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:38.9233357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:39.2774403Z ok (2.145s)
2022-09-27T16:00:39.2774677Z 
2022-09-27T16:00:39.2775156Z ----------------------------------------------------------------------
2022-09-27T16:00:39.2775413Z Ran 1 test in 2.146s
2022-09-27T16:00:39.2775531Z 
2022-09-27T16:00:39.2775591Z OK
2022-09-27T16:00:39.2775683Z 
2022-09-27T16:00:39.2775776Z Generating XML reports...
2022-09-27T16:00:39.2819772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160037.xml
2022-09-27T16:00:40.6069147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:40.6069782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:40.6077543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:40.6078150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:40.9464328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:40.9474495Z 
2022-09-27T16:00:40.9474905Z Running tests...
2022-09-27T16:00:40.9475539Z ----------------------------------------------------------------------
2022-09-27T16:00:41.1911109Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26446
2022-09-27T16:00:41.1937515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26447
2022-09-27T16:00:41.1966106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26448
2022-09-27T16:00:42.2890085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:42.2890701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:42.3143132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:42.3143566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:42.3205378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:42.3205857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:42.3216295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:42.3216851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:42.3492858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:42.3493247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:42.3552358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:42.3552818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:42.6730583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:42.6750099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:42.6750534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:42.6751326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:42.6751849Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:42.6831490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:42.6858200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:42.6858634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:42.7841246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:43.1016054Z ok (2.154s)
2022-09-27T16:00:43.1016259Z 
2022-09-27T16:00:43.1016649Z ----------------------------------------------------------------------
2022-09-27T16:00:43.1016954Z Ran 1 test in 2.154s
2022-09-27T16:00:43.1017056Z 
2022-09-27T16:00:43.1017117Z OK
2022-09-27T16:00:43.1017209Z 
2022-09-27T16:00:43.1017309Z Generating XML reports...
2022-09-27T16:00:43.1066061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160040.xml
2022-09-27T16:00:44.4121739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:44.4122245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:44.4129542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:44.4130002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:44.7503820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:44.7513520Z 
2022-09-27T16:00:44.7513985Z Running tests...
2022-09-27T16:00:44.7514400Z ----------------------------------------------------------------------
2022-09-27T16:00:44.9866849Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26499
2022-09-27T16:00:44.9893073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26500
2022-09-27T16:00:44.9919536Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26501
2022-09-27T16:00:46.0091605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:46.0092293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:46.0164318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:46.0164845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:46.0434342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:46.0434976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:46.0506538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:46.0506983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:46.0615152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:46.0615745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:46.0930223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:46.0930908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:46.4491981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:46.4593209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:46.4593682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:46.4594302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:46.4594820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:46.4595345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:46.4699830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:46.5603575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:46.5604656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:47.0970160Z ok (2.345s)
2022-09-27T16:00:47.0970524Z 
2022-09-27T16:00:47.0971053Z ----------------------------------------------------------------------
2022-09-27T16:00:47.0971352Z Ran 1 test in 2.346s
2022-09-27T16:00:47.0971475Z 
2022-09-27T16:00:47.0971541Z OK
2022-09-27T16:00:47.0971636Z 
2022-09-27T16:00:47.0971717Z Generating XML reports...
2022-09-27T16:00:47.1017069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160044.xml
2022-09-27T16:00:48.3954154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:48.3954541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:48.3962227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:48.3962752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:48.7345339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:48.7354801Z 
2022-09-27T16:00:48.7354941Z Running tests...
2022-09-27T16:00:48.7355558Z ----------------------------------------------------------------------
2022-09-27T16:00:48.9720143Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26552
2022-09-27T16:00:48.9748524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26553
2022-09-27T16:00:48.9774090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26554
2022-09-27T16:00:50.0798662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:50.0799255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:50.1032342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:50.1032735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:50.1122544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:50.1122901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.1237687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:50.1238178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:50.1375930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:50.1376312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.1563890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:50.1564455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.4713239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:50.4814065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:50.4814792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:50.4815449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:50.4816001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:50.4816529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:50.4920003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:50.5826700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:50.5827102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:50.5931706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:50.6033233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:50.6033735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:50.6034412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:50.6034949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:50.6035467Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:50.9827052Z ok (2.247s)
2022-09-27T16:00:50.9827369Z 
2022-09-27T16:00:50.9827879Z ----------------------------------------------------------------------
2022-09-27T16:00:50.9828157Z Ran 1 test in 2.247s
2022-09-27T16:00:50.9828279Z 
2022-09-27T16:00:50.9828344Z OK
2022-09-27T16:00:50.9828437Z 
2022-09-27T16:00:50.9828519Z Generating XML reports...
2022-09-27T16:00:50.9873192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160048.xml
2022-09-27T16:00:52.3260068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:52.3260450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:52.3268551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:52.3269063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:52.6652014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:52.6662286Z 
2022-09-27T16:00:52.6662551Z Running tests...
2022-09-27T16:00:52.6663135Z ----------------------------------------------------------------------
2022-09-27T16:00:52.9024028Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26614
2022-09-27T16:00:52.9050126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26615
2022-09-27T16:00:52.9077162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26616
2022-09-27T16:00:53.9552317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:53.9552928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:53.9805498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:53.9805851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:53.9877302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:53.9877654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:53.9995229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:53.9995590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:54.0155773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:54.0156135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:54.0327087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:54.0327452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:54.3697374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:54.3798599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:54.3799127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:54.3799838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:54.3800365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:54.3800905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:54.3807078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:54.3808133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:54.3809067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:00:54.3809738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:54.4015599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:00:54.4016206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:00:54.4017040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:54.4017886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:54.4114004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:00:54.8126207Z ok (2.146s)
2022-09-27T16:00:54.8127422Z 
2022-09-27T16:00:54.8127738Z ----------------------------------------------------------------------
2022-09-27T16:00:54.8127981Z Ran 1 test in 2.146s
2022-09-27T16:00:54.8128099Z 
2022-09-27T16:00:54.8128164Z OK
2022-09-27T16:00:54.8128256Z 
2022-09-27T16:00:54.8128348Z Generating XML reports...
2022-09-27T16:00:54.8173578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160052.xml
2022-09-27T16:00:56.1160433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:56.1161075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:56.1168214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:56.1168717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:56.4557838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:56.4567035Z 
2022-09-27T16:00:56.4567325Z Running tests...
2022-09-27T16:00:56.4567676Z ----------------------------------------------------------------------
2022-09-27T16:00:56.4594571Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-09-27T16:00:56.4594879Z 
2022-09-27T16:00:56.4595111Z ----------------------------------------------------------------------
2022-09-27T16:00:56.4595364Z Ran 1 test in 0.003s
2022-09-27T16:00:56.4595466Z 
2022-09-27T16:00:56.4595547Z OK (skipped=1)
2022-09-27T16:00:56.4595657Z 
2022-09-27T16:00:56.4595744Z Generating XML reports...
2022-09-27T16:00:56.4625305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160056.xml
2022-09-27T16:00:57.6515645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.6516040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.6523646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.6524043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:58.0026387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:58.0034779Z 
2022-09-27T16:00:58.0034919Z Running tests...
2022-09-27T16:00:58.0035363Z ----------------------------------------------------------------------
2022-09-27T16:00:58.2381670Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26683
2022-09-27T16:00:58.2408086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26684
2022-09-27T16:00:58.2435506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26685
2022-09-27T16:00:59.3315496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:59.3315871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:59.3452875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:59.3453424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:59.3654803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:59.3655433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:59.3695684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:59.3696286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:59.3799879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:59.3800523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:59.4021971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:59.4022596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:59.7407477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:59.7408244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:59.7408779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:00:59.7409789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:59.7410469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:59.7411005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:00:59.7512882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:59.8419908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:59.8420492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:00.2485331Z ok (2.245s)
2022-09-27T16:01:00.2485581Z 
2022-09-27T16:01:00.2486031Z ----------------------------------------------------------------------
2022-09-27T16:01:00.2486311Z Ran 1 test in 2.245s
2022-09-27T16:01:00.2486430Z 
2022-09-27T16:01:00.2486493Z OK
2022-09-27T16:01:00.2486587Z 
2022-09-27T16:01:00.2486668Z Generating XML reports...
2022-09-27T16:01:00.2531122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160057.xml
2022-09-27T16:01:01.5738450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:01.5738820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:01.5745932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:01.5746297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:01.9174206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:01.9183470Z 
2022-09-27T16:01:01.9183607Z Running tests...
2022-09-27T16:01:01.9184183Z ----------------------------------------------------------------------
2022-09-27T16:01:02.1552653Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26736
2022-09-27T16:01:02.1579654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26737
2022-09-27T16:01:02.1606850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26738
2022-09-27T16:01:03.2245554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:03.2246247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:03.2555395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:03.2556043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:03.2691001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:03.2691629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:03.2777999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:03.2778579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:03.3032642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:03.3033552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:03.3120599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:03.3121572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:03.6345768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:03.6446781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:03.6447548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:03.6448781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:03.6449387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:03.6449935Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:03.6457031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:03.6457699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:03.6458183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:04.0657877Z ok (2.147s)
2022-09-27T16:01:04.0658078Z 
2022-09-27T16:01:04.0658400Z ----------------------------------------------------------------------
2022-09-27T16:01:04.0658654Z Ran 1 test in 2.147s
2022-09-27T16:01:04.0658775Z 
2022-09-27T16:01:04.0658838Z OK
2022-09-27T16:01:04.0658931Z 
2022-09-27T16:01:04.0659025Z Generating XML reports...
2022-09-27T16:01:04.0704425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160101.xml
2022-09-27T16:01:05.3920690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:05.3921064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:05.3928624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:05.3928970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:05.7324544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:05.7334874Z 
2022-09-27T16:01:05.7335255Z Running tests...
2022-09-27T16:01:05.7335641Z ----------------------------------------------------------------------
2022-09-27T16:01:05.9685948Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26789
2022-09-27T16:01:05.9712963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26790
2022-09-27T16:01:05.9739655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26791
2022-09-27T16:01:07.0756333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:07.0757060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:07.1059651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:07.1060352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:07.1076856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:07.1077524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:07.1111336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:07.1112187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:07.1405372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:07.1406339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:07.1452796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:07.1453437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:07.4771647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:07.4872228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:07.4872703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:07.4873384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:07.4873929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:07.4874473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:07.4979930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:07.4980525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:07.4981590Z STAGE:2022-09-27 16:01:07 26790:26790 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:01:07.4982303Z STAGE:2022-09-27 16:01:07 26791:26791 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:01:07.5882352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:07.5884405Z STAGE:2022-09-27 16:01:07 26789:26789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:01:07.5921673Z STAGE:2022-09-27 16:01:07 26789:26789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:07.5923271Z STAGE:2022-09-27 16:01:07 26791:26791 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:07.5924446Z STAGE:2022-09-27 16:01:07 26790:26790 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:07.9792778Z ok (2.245s)
2022-09-27T16:01:07.9792992Z 
2022-09-27T16:01:07.9793532Z ----------------------------------------------------------------------
2022-09-27T16:01:07.9793887Z Ran 1 test in 2.246s
2022-09-27T16:01:07.9794003Z 
2022-09-27T16:01:07.9794064Z OK
2022-09-27T16:01:07.9794155Z 
2022-09-27T16:01:07.9794250Z Generating XML reports...
2022-09-27T16:01:07.9838605Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160105.xml
2022-09-27T16:01:09.2977050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:09.2977433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:09.2985349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:09.2985884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:09.6368582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:09.6377850Z 
2022-09-27T16:01:09.6377997Z Running tests...
2022-09-27T16:01:09.6378602Z ----------------------------------------------------------------------
2022-09-27T16:01:09.8731235Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26845
2022-09-27T16:01:09.8758222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26846
2022-09-27T16:01:09.8784412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26847
2022-09-27T16:01:10.9330884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:10.9331485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:10.9445796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:10.9446149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:10.9564974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:10.9565517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:10.9680373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:10.9681036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:10.9806033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:10.9806677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:10.9912863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:10.9913503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:11.3309769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:11.3310392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:11.3310794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:11.3311526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:11.3312449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:11.3313317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:11.3317795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:11.3318342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:11.3322543Z STAGE:2022-09-27 16:01:11 26847:26847 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:01:11 26846:26846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:01:11.3322986Z 
2022-09-27T16:01:11.4317891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:11.4322403Z STAGE:2022-09-27 16:01:11 26845:26845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:01:11.4353409Z STAGE:2022-09-27 16:01:11 26847:26847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:11.4354213Z STAGE:2022-09-27 16:01:11 26846:26846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:11.5339662Z STAGE:2022-09-27 16:01:11 26845:26845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:01:11.8836259Z ok (2.245s)
2022-09-27T16:01:11.8836490Z 
2022-09-27T16:01:11.8837004Z ----------------------------------------------------------------------
2022-09-27T16:01:11.8837390Z Ran 1 test in 2.246s
2022-09-27T16:01:11.8837507Z 
2022-09-27T16:01:11.8837570Z OK
2022-09-27T16:01:11.8837651Z 
2022-09-27T16:01:11.8837742Z Generating XML reports...
2022-09-27T16:01:11.8885470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160109.xml
2022-09-27T16:01:13.1961471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:13.1961837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:13.1969211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:13.1969576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:13.5364652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:13.5373701Z 
2022-09-27T16:01:13.5373824Z Running tests...
2022-09-27T16:01:13.5374704Z ----------------------------------------------------------------------
2022-09-27T16:01:13.5391281Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T16:01:13.5391766Z 
2022-09-27T16:01:13.5392031Z ----------------------------------------------------------------------
2022-09-27T16:01:13.5392267Z Ran 1 test in 0.002s
2022-09-27T16:01:13.5392382Z 
2022-09-27T16:01:13.5392457Z OK (skipped=1)
2022-09-27T16:01:13.5392572Z 
2022-09-27T16:01:13.5392660Z Generating XML reports...
2022-09-27T16:01:13.5421128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160113.xml
2022-09-27T16:01:14.7494241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:14.7494640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:14.7503114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:14.7503704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:15.1053497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:15.1063505Z 
2022-09-27T16:01:15.1063613Z Running tests...
2022-09-27T16:01:15.1064427Z ----------------------------------------------------------------------
2022-09-27T16:01:15.1081261Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T16:01:15.1081751Z 
2022-09-27T16:01:15.1082115Z ----------------------------------------------------------------------
2022-09-27T16:01:15.1082354Z Ran 1 test in 0.002s
2022-09-27T16:01:15.1082471Z 
2022-09-27T16:01:15.1082544Z OK (skipped=1)
2022-09-27T16:01:15.1082654Z 
2022-09-27T16:01:15.1082744Z Generating XML reports...
2022-09-27T16:01:15.1114145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160115.xml
2022-09-27T16:01:16.2956593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:16.2956992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:16.2964615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:16.2965000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:16.6360542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:16.6370408Z 
2022-09-27T16:01:16.6370870Z Running tests...
2022-09-27T16:01:16.6371263Z ----------------------------------------------------------------------
2022-09-27T16:01:16.8713197Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26921
2022-09-27T16:01:16.8740185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26922
2022-09-27T16:01:16.8770717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26923
2022-09-27T16:01:17.9712338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:17.9713306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:18.0028148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:18.0028762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:18.0048067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:18.0048691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:18.0073210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:18.0073807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:18.0397536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:18.0398195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:18.0425821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:18.0426441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:18.3771361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:18.3872828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:18.3873492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:18.3874294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:18.3874911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:18.3875799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:18.3979333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:18.4884131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:18.4884612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:20.4908774Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:01:20.5007109Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:01:20.8846283Z ok (4.247s)
2022-09-27T16:01:20.8846528Z 
2022-09-27T16:01:20.8847094Z ----------------------------------------------------------------------
2022-09-27T16:01:20.8847437Z Ran 1 test in 4.248s
2022-09-27T16:01:20.8847561Z 
2022-09-27T16:01:20.8847610Z OK
2022-09-27T16:01:20.8847703Z 
2022-09-27T16:01:20.8847797Z Generating XML reports...
2022-09-27T16:01:20.8897250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160116.xml
2022-09-27T16:01:22.2158994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:22.2159460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:22.2166468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:22.2166836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:22.5550146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:22.5558943Z 
2022-09-27T16:01:22.5559464Z Running tests...
2022-09-27T16:01:22.5559871Z ----------------------------------------------------------------------
2022-09-27T16:01:22.7949714Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26974
2022-09-27T16:01:22.7976322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26975
2022-09-27T16:01:22.8002679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26976
2022-09-27T16:01:23.9030830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:23.9031203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:23.9053783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:23.9054457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:23.9122597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:23.9122969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:23.9385220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:23.9385625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:23.9410088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:23.9410626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:23.9481209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:23.9481730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:24.2886864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:24.2988180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:24.2988832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:24.2989741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:24.2990301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:24.2991152Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:24.2996898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:24.2998862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:24.2999397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:26.3197089Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:01:26.3197575Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:01:26.7076677Z ok (4.151s)
2022-09-27T16:01:26.7076930Z 
2022-09-27T16:01:26.7077398Z ----------------------------------------------------------------------
2022-09-27T16:01:26.7077794Z Ran 1 test in 4.152s
2022-09-27T16:01:26.7077987Z 
2022-09-27T16:01:26.7078086Z OK
2022-09-27T16:01:26.7078244Z 
2022-09-27T16:01:26.7078390Z Generating XML reports...
2022-09-27T16:01:26.7125259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160122.xml
2022-09-27T16:01:28.0079830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:28.0080451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:28.0087917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:28.0088277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:28.3463623Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:28.3473093Z 
2022-09-27T16:01:28.3473383Z Running tests...
2022-09-27T16:01:28.3473859Z ----------------------------------------------------------------------
2022-09-27T16:01:28.5825804Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27027
2022-09-27T16:01:28.5852978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27028
2022-09-27T16:01:28.5879928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27029
2022-09-27T16:01:29.6709912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:29.6710577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:29.7029406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:29.7030003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:29.7080173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:29.7080778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:29.7090883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:29.7091478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:29.7432213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:29.7432623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:29.7441519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:29.7442098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:30.0909811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:30.1010749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:30.1011432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:30.1012293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:30.1012838Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:30.1013350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:30.1018623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:30.1019919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:30.1020559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:30.1228568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:01:30.1229412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:01:30.1230035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:01:30.1230962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:30.1231746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:30.1232484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:30.1233338Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-09-27T16:01:30.1233682Z No ranks successfully processed in monitoredBarrier.
2022-09-27T16:01:30.1260677Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-09-27T16:01:30.4932463Z ok (2.145s)
2022-09-27T16:01:30.4932654Z 
2022-09-27T16:01:30.4932979Z ----------------------------------------------------------------------
2022-09-27T16:01:30.4933235Z Ran 1 test in 2.146s
2022-09-27T16:01:30.4933358Z 
2022-09-27T16:01:30.4933435Z OK
2022-09-27T16:01:30.4933528Z 
2022-09-27T16:01:30.4933622Z Generating XML reports...
2022-09-27T16:01:30.4978176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160128.xml
2022-09-27T16:01:31.8283088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:31.8283449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:31.8291180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:31.8291540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:32.1677602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:32.1688238Z 
2022-09-27T16:01:32.1688641Z Running tests...
2022-09-27T16:01:32.1689038Z ----------------------------------------------------------------------
2022-09-27T16:01:32.4044609Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27089
2022-09-27T16:01:32.4070477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27090
2022-09-27T16:01:32.4097683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27091
2022-09-27T16:01:33.4800931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.4801547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.4839684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.4840306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.5041001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.5041438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.5149514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.5149892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.5198843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.5199397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.5382192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.5382860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.8629631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:33.8730066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:33.8730732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:33.8731666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:33.8732229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:33.8733067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:33.8738295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:33.8738948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:33.8739954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:33.8741697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:01:33.8844978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:01:33.8845659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:01:33.8846355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:33.8846892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:33.8946757Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:01:33.8951793Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:350: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-09-27T16:01:33.8952218Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-09-27T16:01:33.9951630Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:01:34.4147727Z ok (2.246s)
2022-09-27T16:01:34.4147928Z 
2022-09-27T16:01:34.4148359Z ----------------------------------------------------------------------
2022-09-27T16:01:34.4148824Z Ran 1 test in 2.246s
2022-09-27T16:01:34.4149028Z 
2022-09-27T16:01:34.4149106Z OK
2022-09-27T16:01:34.4149201Z 
2022-09-27T16:01:34.4149296Z Generating XML reports...
2022-09-27T16:01:34.4195446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160132.xml
2022-09-27T16:01:35.7186638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:35.7187025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:35.7194851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:35.7195226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:36.0548808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:36.0559057Z 
2022-09-27T16:01:36.0559606Z Running tests...
2022-09-27T16:01:36.0560011Z ----------------------------------------------------------------------
2022-09-27T16:01:36.2913268Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27148
2022-09-27T16:01:36.2939663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27149
2022-09-27T16:01:36.2966768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27150
2022-09-27T16:01:37.3812961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:37.3813938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:37.4132418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:37.4132828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:37.4209740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:37.4210098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:37.4531036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:37.4531493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:37.4605398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:37.4605960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:37.4919678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:37.4920127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:37.8139832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:37.8240193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:37.8240822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:37.8241737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:37.8242322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:37.8242853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:37.8248967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:37.8249669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:37.9250906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:38.0451544Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:01:38.0452030Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:01:38.0452426Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:01:38.4019390Z ok (2.346s)
2022-09-27T16:01:38.4019646Z 
2022-09-27T16:01:38.4020216Z ----------------------------------------------------------------------
2022-09-27T16:01:38.4020592Z Ran 1 test in 2.346s
2022-09-27T16:01:38.4020709Z 
2022-09-27T16:01:38.4020756Z OK
2022-09-27T16:01:38.4020850Z 
2022-09-27T16:01:38.4020942Z Generating XML reports...
2022-09-27T16:01:38.4065251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160136.xml
2022-09-27T16:01:39.7106239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:39.7106623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:39.7114152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:39.7114774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:40.0466800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:40.0476155Z 
2022-09-27T16:01:40.0476299Z Running tests...
2022-09-27T16:01:40.0476768Z ----------------------------------------------------------------------
2022-09-27T16:01:40.0497724Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:01:40.0497985Z 
2022-09-27T16:01:40.0498253Z ----------------------------------------------------------------------
2022-09-27T16:01:40.0498590Z Ran 1 test in 0.002s
2022-09-27T16:01:40.0498697Z 
2022-09-27T16:01:40.0498774Z OK (skipped=1)
2022-09-27T16:01:40.0498883Z 
2022-09-27T16:01:40.0498972Z Generating XML reports...
2022-09-27T16:01:40.0528929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160140.xml
2022-09-27T16:01:41.2430359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:41.2430992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:41.2439649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:41.2440294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:41.5837839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:41.5848061Z 
2022-09-27T16:01:41.5848345Z Running tests...
2022-09-27T16:01:41.5848948Z ----------------------------------------------------------------------
2022-09-27T16:01:41.5872120Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:01:41.5872568Z 
2022-09-27T16:01:41.5872816Z ----------------------------------------------------------------------
2022-09-27T16:01:41.5873056Z Ran 1 test in 0.002s
2022-09-27T16:01:41.5873171Z 
2022-09-27T16:01:41.5873247Z OK (skipped=1)
2022-09-27T16:01:41.5873365Z 
2022-09-27T16:01:41.5873450Z Generating XML reports...
2022-09-27T16:01:41.5902010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160141.xml
2022-09-27T16:01:42.7850800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:42.7851431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:42.7859497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:42.7860165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:43.1259893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:43.1270496Z 
2022-09-27T16:01:43.1270923Z Running tests...
2022-09-27T16:01:43.1271341Z ----------------------------------------------------------------------
2022-09-27T16:01:43.1291779Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:01:43.1292201Z 
2022-09-27T16:01:43.1292534Z ----------------------------------------------------------------------
2022-09-27T16:01:43.1292925Z Ran 1 test in 0.002s
2022-09-27T16:01:43.1293118Z 
2022-09-27T16:01:43.1293241Z OK (skipped=1)
2022-09-27T16:01:43.1293433Z 
2022-09-27T16:01:43.1293578Z Generating XML reports...
2022-09-27T16:01:43.1323888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160143.xml
2022-09-27T16:01:44.3209306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:44.3210240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:44.3218152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:44.3219002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:44.6595460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:44.6605805Z 
2022-09-27T16:01:44.6606067Z Running tests...
2022-09-27T16:01:44.6606670Z ----------------------------------------------------------------------
2022-09-27T16:01:44.6632702Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:01:44.6633012Z 
2022-09-27T16:01:44.6633225Z ----------------------------------------------------------------------
2022-09-27T16:01:44.6633512Z Ran 1 test in 0.003s
2022-09-27T16:01:44.6633638Z 
2022-09-27T16:01:44.6633712Z OK (skipped=1)
2022-09-27T16:01:44.6633820Z 
2022-09-27T16:01:44.6633913Z Generating XML reports...
2022-09-27T16:01:44.6662828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160144.xml
2022-09-27T16:01:45.8800440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:45.8801096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:45.8809307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:45.8809695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:46.2180914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:46.2190543Z 
2022-09-27T16:01:46.2190684Z Running tests...
2022-09-27T16:01:46.2191126Z ----------------------------------------------------------------------
2022-09-27T16:01:46.2212378Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T16:01:46.2212773Z 
2022-09-27T16:01:46.2213089Z ----------------------------------------------------------------------
2022-09-27T16:01:46.2213337Z Ran 1 test in 0.002s
2022-09-27T16:01:46.2213453Z 
2022-09-27T16:01:46.2213547Z OK (skipped=1)
2022-09-27T16:01:46.2213683Z 
2022-09-27T16:01:46.2213771Z Generating XML reports...
2022-09-27T16:01:46.2242770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160146.xml
2022-09-27T16:01:47.4152567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:47.4152949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:47.4160076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:47.4160554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:47.7538563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:47.7548241Z 
2022-09-27T16:01:47.7548384Z Running tests...
2022-09-27T16:01:47.7548980Z ----------------------------------------------------------------------
2022-09-27T16:01:47.7568233Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:01:47.7568571Z 
2022-09-27T16:01:47.7568894Z ----------------------------------------------------------------------
2022-09-27T16:01:47.7569127Z Ran 1 test in 0.002s
2022-09-27T16:01:47.7569241Z 
2022-09-27T16:01:47.7569318Z OK (skipped=1)
2022-09-27T16:01:47.7569427Z 
2022-09-27T16:01:47.7569512Z Generating XML reports...
2022-09-27T16:01:47.7599177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160147.xml
2022-09-27T16:01:48.9735587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:48.9735973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:48.9743408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:48.9743776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:49.3137777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:49.3147352Z 
2022-09-27T16:01:49.3147769Z Running tests...
2022-09-27T16:01:49.3148195Z ----------------------------------------------------------------------
2022-09-27T16:01:49.3169507Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:01:49.3169891Z 
2022-09-27T16:01:49.3170311Z ----------------------------------------------------------------------
2022-09-27T16:01:49.3170718Z Ran 1 test in 0.002s
2022-09-27T16:01:49.3170832Z 
2022-09-27T16:01:49.3170906Z OK (skipped=1)
2022-09-27T16:01:49.3171023Z 
2022-09-27T16:01:49.3171107Z Generating XML reports...
2022-09-27T16:01:49.3200731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160149.xml
2022-09-27T16:01:50.5044767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:50.5045139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:50.5053169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:50.5053565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:50.8435420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:50.8445939Z 
2022-09-27T16:01:50.8446212Z Running tests...
2022-09-27T16:01:50.8446653Z ----------------------------------------------------------------------
2022-09-27T16:01:50.8465839Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:01:50.8466204Z 
2022-09-27T16:01:50.8466452Z ----------------------------------------------------------------------
2022-09-27T16:01:50.8466754Z Ran 1 test in 0.002s
2022-09-27T16:01:50.8466871Z 
2022-09-27T16:01:50.8466943Z OK (skipped=1)
2022-09-27T16:01:50.8467052Z 
2022-09-27T16:01:50.8467139Z Generating XML reports...
2022-09-27T16:01:50.8496342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160150.xml
2022-09-27T16:01:52.0658545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:52.0659198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:52.0667481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:52.0668139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:52.4069444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:52.4078871Z 
2022-09-27T16:01:52.4079182Z Running tests...
2022-09-27T16:01:52.4079789Z ----------------------------------------------------------------------
2022-09-27T16:01:52.6470556Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27281
2022-09-27T16:01:52.6498004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27282
2022-09-27T16:01:52.6525502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27283
2022-09-27T16:01:53.7632579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:53.7633215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:53.7735440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:53.7736091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:53.7940569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:53.7941271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:53.7973854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:53.7974548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:53.8088885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:53.8089462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:53.8273817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:53.8274445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:54.1739752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:54.1740208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:54.1740581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:54.1741269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:54.1741884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:54.1742433Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:54.1846028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:54.2752012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:54.2752469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:54.5576338Z skip: CUDA is not available. (2.149s)
2022-09-27T16:01:54.5576553Z 
2022-09-27T16:01:54.5576867Z ----------------------------------------------------------------------
2022-09-27T16:01:54.5577124Z Ran 1 test in 2.150s
2022-09-27T16:01:54.5577241Z 
2022-09-27T16:01:54.5577317Z OK (skipped=1)
2022-09-27T16:01:54.5577427Z 
2022-09-27T16:01:54.5577515Z Generating XML reports...
2022-09-27T16:01:54.5623381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160152.xml
2022-09-27T16:01:55.8896235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:55.8896876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:55.8904863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:55.8905506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:56.2272392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:56.2282310Z 
2022-09-27T16:01:56.2282760Z Running tests...
2022-09-27T16:01:56.2283162Z ----------------------------------------------------------------------
2022-09-27T16:01:56.4632713Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27334
2022-09-27T16:01:56.4658709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27335
2022-09-27T16:01:56.4685561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27336
2022-09-27T16:01:57.5256976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:57.5257645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:57.5382193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:57.5382679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:57.5588424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:57.5589083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:57.5724423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:57.5725073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:57.5875905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:57.5876532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:57.6210203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:57.6210825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:57.9602678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:57.9702602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:01:57.9703242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:57.9704339Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:57.9705279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:57.9706170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:01:57.9810128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:01:58.0716981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:58.0717407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:58.3736815Z skip: CUDA is not available. (2.145s)
2022-09-27T16:01:58.3737200Z 
2022-09-27T16:01:58.3737897Z ----------------------------------------------------------------------
2022-09-27T16:01:58.3738381Z Ran 1 test in 2.145s
2022-09-27T16:01:58.3738617Z 
2022-09-27T16:01:58.3738740Z OK (skipped=1)
2022-09-27T16:01:58.3738940Z 
2022-09-27T16:01:58.3739069Z Generating XML reports...
2022-09-27T16:01:58.3784848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160156.xml
2022-09-27T16:01:59.6758476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:59.6758847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:59.6766658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:59.6767029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:00.0156349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:00.0165415Z 
2022-09-27T16:02:00.0165537Z Running tests...
2022-09-27T16:02:00.0166042Z ----------------------------------------------------------------------
2022-09-27T16:02:00.0183247Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:02:00.0183609Z 
2022-09-27T16:02:00.0184227Z ----------------------------------------------------------------------
2022-09-27T16:02:00.0184596Z Ran 1 test in 0.002s
2022-09-27T16:02:00.0184773Z 
2022-09-27T16:02:00.0184850Z OK (skipped=1)
2022-09-27T16:02:00.0184959Z 
2022-09-27T16:02:00.0185046Z Generating XML reports...
2022-09-27T16:02:00.0213831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160200.xml
2022-09-27T16:02:01.2087311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:01.2087696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:01.2094977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:01.2095358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:01.5476012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:01.5485178Z 
2022-09-27T16:02:01.5485297Z Running tests...
2022-09-27T16:02:01.5485735Z ----------------------------------------------------------------------
2022-09-27T16:02:01.5502823Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s)
2022-09-27T16:02:01.5503304Z 
2022-09-27T16:02:01.5503720Z ----------------------------------------------------------------------
2022-09-27T16:02:01.5504139Z Ran 1 test in 0.002s
2022-09-27T16:02:01.5504271Z 
2022-09-27T16:02:01.5504346Z OK (skipped=1)
2022-09-27T16:02:01.5504459Z 
2022-09-27T16:02:01.5504545Z Generating XML reports...
2022-09-27T16:02:01.5533163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160201.xml
2022-09-27T16:02:02.7798350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:02.7798737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:02.7806152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:02.7806519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:03.1443399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:03.1452945Z 
2022-09-27T16:02:03.1453080Z Running tests...
2022-09-27T16:02:03.1453705Z ----------------------------------------------------------------------
2022-09-27T16:02:03.3773978Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.232s)
2022-09-27T16:02:03.3774553Z 
2022-09-27T16:02:03.3774750Z ----------------------------------------------------------------------
2022-09-27T16:02:03.3774994Z Ran 1 test in 0.232s
2022-09-27T16:02:03.3775111Z 
2022-09-27T16:02:03.3775181Z OK (skipped=1)
2022-09-27T16:02:03.3775289Z 
2022-09-27T16:02:03.3775374Z Generating XML reports...
2022-09-27T16:02:03.3797129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160203.xml
2022-09-27T16:02:04.6240974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:04.6241389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:04.6248921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:04.6249287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:04.9632154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:04.9641137Z 
2022-09-27T16:02:04.9641236Z Running tests...
2022-09-27T16:02:04.9641693Z ----------------------------------------------------------------------
2022-09-27T16:02:05.2006063Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27417
2022-09-27T16:02:05.2032558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27418
2022-09-27T16:02:05.2059441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27419
2022-09-27T16:02:06.2813812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:06.2814204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:06.3133926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:06.3134547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:06.3234699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:06.3235323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:06.3290836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:06.3291475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:06.3577682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:06.3578334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:06.3637128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:06.3637769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:06.7057097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:06.7158380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:06.7159058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:06.7159882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:06.7160476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:06.7161016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:06.7266799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:06.8171448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:06.8270207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:07.1107776Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T16:02:07.1108066Z 
2022-09-27T16:02:07.1108552Z ----------------------------------------------------------------------
2022-09-27T16:02:07.1109038Z Ran 1 test in 2.147s
2022-09-27T16:02:07.1109242Z 
2022-09-27T16:02:07.1109523Z OK (skipped=1)
2022-09-27T16:02:07.1109638Z 
2022-09-27T16:02:07.1109711Z Generating XML reports...
2022-09-27T16:02:07.1154330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160204.xml
2022-09-27T16:02:08.4156450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:08.4156815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:08.4164956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:08.4165501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:08.7547152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:08.7556665Z 
2022-09-27T16:02:08.7557056Z Running tests...
2022-09-27T16:02:08.7557676Z ----------------------------------------------------------------------
2022-09-27T16:02:08.9920070Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27470
2022-09-27T16:02:08.9946169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27471
2022-09-27T16:02:08.9973159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27472
2022-09-27T16:02:10.1426111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:10.1426712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:10.1476183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:10.1476836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:10.1493923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:10.1494582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:10.1779140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:10.1779857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:10.1840501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:10.1841125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:10.1855726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:10.1856382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:10.5109764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:10.5110352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:10.5110987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:10.5111387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:10.5111885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:10.5112403Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:10.5214990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:10.6121268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:10.6121953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:10.9024016Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T16:02:10.9024281Z 
2022-09-27T16:02:10.9024864Z ----------------------------------------------------------------------
2022-09-27T16:02:10.9025386Z Ran 1 test in 2.147s
2022-09-27T16:02:10.9025502Z 
2022-09-27T16:02:10.9025562Z OK (skipped=1)
2022-09-27T16:02:10.9025671Z 
2022-09-27T16:02:10.9025759Z Generating XML reports...
2022-09-27T16:02:10.9072595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160208.xml
2022-09-27T16:02:12.2167906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:12.2168272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:12.2176338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:12.2176711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:12.5557274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:12.5566519Z 
2022-09-27T16:02:12.5566763Z Running tests...
2022-09-27T16:02:12.5567175Z ----------------------------------------------------------------------
2022-09-27T16:02:12.7967918Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27523
2022-09-27T16:02:12.7995556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27524
2022-09-27T16:02:12.8022227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27525
2022-09-27T16:02:13.9128308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:13.9128916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:13.9400069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:13.9400668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:13.9445377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:13.9445969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:13.9524526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:13.9525178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:13.9742436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:13.9743013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:13.9871062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:13.9871461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:14.3146989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:14.3147543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:14.3148213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:14.3148630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:14.3149129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:14.3149639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:14.3252920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:14.3253699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:14.4160677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:14.7072594Z skip: Need at least 2 CUDA devices (2.150s)
2022-09-27T16:02:14.7072861Z 
2022-09-27T16:02:14.7073376Z ----------------------------------------------------------------------
2022-09-27T16:02:14.7073778Z Ran 1 test in 2.151s
2022-09-27T16:02:14.7073895Z 
2022-09-27T16:02:14.7073969Z OK (skipped=1)
2022-09-27T16:02:14.7074078Z 
2022-09-27T16:02:14.7074151Z Generating XML reports...
2022-09-27T16:02:14.7119783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160212.xml
2022-09-27T16:02:16.0388613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:16.0388994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:16.0396809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:16.0397179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:16.3764449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:16.3774048Z 
2022-09-27T16:02:16.3774235Z Running tests...
2022-09-27T16:02:16.3774793Z ----------------------------------------------------------------------
2022-09-27T16:02:16.6125868Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27576
2022-09-27T16:02:16.6153831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27577
2022-09-27T16:02:16.6180497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27578
2022-09-27T16:02:17.6766754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.6767441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.7089195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.7089874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.7426868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.7427528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.7728973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.7729633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.7742919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.7743310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.8049843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.8050451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:18.1395563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:18.1517481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:18.1518407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:18.1520035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:18.1520966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:18.1521943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:18.1522585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:18.1523166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:18.1523705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:18.5228581Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T16:02:18.5228869Z 
2022-09-27T16:02:18.5229402Z ----------------------------------------------------------------------
2022-09-27T16:02:18.5229695Z Ran 1 test in 2.145s
2022-09-27T16:02:18.5229815Z 
2022-09-27T16:02:18.5229894Z OK (skipped=1)
2022-09-27T16:02:18.5230020Z 
2022-09-27T16:02:18.5230107Z Generating XML reports...
2022-09-27T16:02:18.5277187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160216.xml
2022-09-27T16:02:19.8305259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:19.8305903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:19.8313879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:19.8314538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:20.1674987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:20.1685035Z 
2022-09-27T16:02:20.1685342Z Running tests...
2022-09-27T16:02:20.1685760Z ----------------------------------------------------------------------
2022-09-27T16:02:20.4033578Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27629
2022-09-27T16:02:20.4062509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27630
2022-09-27T16:02:20.4089457Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27631
2022-09-27T16:02:21.4777628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:21.4778329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:21.4845795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:21.4846330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:21.4966638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:21.4967165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:21.5141526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:21.5142178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:21.5219339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:21.5219948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:21.5354968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:21.5355647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:21.8695774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:21.8696714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:21.8697075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:21.8697784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:21.8698316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:21.8698845Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:21.8704172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:21.8705081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:21.8705767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:22.2138072Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T16:02:22.2138322Z 
2022-09-27T16:02:22.2138711Z ----------------------------------------------------------------------
2022-09-27T16:02:22.2139064Z Ran 1 test in 2.045s
2022-09-27T16:02:22.2139367Z 
2022-09-27T16:02:22.2139493Z OK (skipped=1)
2022-09-27T16:02:22.2139604Z 
2022-09-27T16:02:22.2139702Z Generating XML reports...
2022-09-27T16:02:22.2188436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160220.xml
2022-09-27T16:02:23.5134929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:23.5135323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:23.5142356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:23.5142733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:23.8536745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:23.8546430Z 
2022-09-27T16:02:23.8546780Z Running tests...
2022-09-27T16:02:23.8547184Z ----------------------------------------------------------------------
2022-09-27T16:02:24.0906003Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27682
2022-09-27T16:02:24.0932926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27683
2022-09-27T16:02:24.0960296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27684
2022-09-27T16:02:25.1903542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:25.1904401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:25.1905285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:25.1905906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:25.2179451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:25.2180088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:25.2253066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:25.2253697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:25.2254450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:25.2255019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:25.2510135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:25.2510831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:25.5948316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:25.6049277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:25.6049867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:25.6050834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:25.6051666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:25.6052459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:25.6155871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:25.7062178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:25.7062631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:26.0009765Z skip: Need at least 4 CUDA devices (2.146s)
2022-09-27T16:02:26.0009944Z 
2022-09-27T16:02:26.0010237Z ----------------------------------------------------------------------
2022-09-27T16:02:26.0010485Z Ran 1 test in 2.146s
2022-09-27T16:02:26.0010704Z 
2022-09-27T16:02:26.0010780Z OK (skipped=1)
2022-09-27T16:02:26.0010894Z 
2022-09-27T16:02:26.0010980Z Generating XML reports...
2022-09-27T16:02:26.0057960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml
2022-09-27T16:02:27.3086140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:27.3086526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:27.3093953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:27.3094515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:27.6467667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:27.6476630Z 
2022-09-27T16:02:27.6476753Z Running tests...
2022-09-27T16:02:27.6477354Z ----------------------------------------------------------------------
2022-09-27T16:02:27.8865165Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27735
2022-09-27T16:02:27.8890688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27736
2022-09-27T16:02:27.8917247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27737
2022-09-27T16:02:28.9851142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:28.9851845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:28.9894880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:28.9895304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:29.0182476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:29.0182968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:29.0194539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:29.0195159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:29.0254246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:29.0254745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:29.0512136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:29.0512499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:29.3693330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:29.3794877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:29.3795323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:29.3795969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:29.3796815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:29.3797614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:29.3801827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:29.3802401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:29.3804792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:29.6966717Z skip: Need at least 4 CUDA devices (2.049s)
2022-09-27T16:02:29.6966899Z 
2022-09-27T16:02:29.6967303Z ----------------------------------------------------------------------
2022-09-27T16:02:29.6967556Z Ran 1 test in 2.049s
2022-09-27T16:02:29.6967671Z 
2022-09-27T16:02:29.6967756Z OK (skipped=1)
2022-09-27T16:02:29.6967854Z 
2022-09-27T16:02:29.6967940Z Generating XML reports...
2022-09-27T16:02:29.7015923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160227.xml
2022-09-27T16:02:30.9934143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:30.9934560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:30.9942086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:30.9942450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:31.3355903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:31.3365591Z 
2022-09-27T16:02:31.3366214Z Running tests...
2022-09-27T16:02:31.3366653Z ----------------------------------------------------------------------
2022-09-27T16:02:31.5724292Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27788
2022-09-27T16:02:31.5751076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27789
2022-09-27T16:02:31.5778070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27790
2022-09-27T16:02:32.6204329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:32.6205021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:32.6519676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:32.6520318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:32.6576385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:32.6577019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:32.6748834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:32.6749458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:32.6935154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:32.6935529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:32.7096763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:32.7097138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:33.0683968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:33.0784096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:33.0785207Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:33.0785971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:33.0786695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:33.0787223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:33.0891095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:33.1799650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:33.1800340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:33.4826823Z skip: Need at least 2 CUDA devices (2.146s)
2022-09-27T16:02:33.4827116Z 
2022-09-27T16:02:33.4827561Z ----------------------------------------------------------------------
2022-09-27T16:02:33.4827997Z Ran 1 test in 2.146s
2022-09-27T16:02:33.4828155Z 
2022-09-27T16:02:33.4828265Z OK (skipped=1)
2022-09-27T16:02:33.4828432Z 
2022-09-27T16:02:33.4828560Z Generating XML reports...
2022-09-27T16:02:33.4874753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160231.xml
2022-09-27T16:02:34.8286667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:34.8287294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:34.8295300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:34.8295961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:35.1733096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:35.1743182Z 
2022-09-27T16:02:35.1743739Z Running tests...
2022-09-27T16:02:35.1744364Z ----------------------------------------------------------------------
2022-09-27T16:02:35.4117670Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27841
2022-09-27T16:02:35.4145140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27842
2022-09-27T16:02:35.4171962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27843
2022-09-27T16:02:36.5406319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.5406949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.5413664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.5414324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.5498572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.5499074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.5763115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.5763726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.5772144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.5772547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.5852775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.5853233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.9215792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:36.9316799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:36.9317672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:36.9318679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:36.9319217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:36.9319738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:36.9324631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:36.9325542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:36.9328805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:36.9532546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:36.9633918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:36.9634468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:36.9635086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:36.9635671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:36.9636196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:36.9639640Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9639930Z 
2022-09-27T16:02:36.9739498Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9743679Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9744595Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9745310Z STAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9752319Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9754853Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9760569Z STAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9764409Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9765817Z STAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9766479Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9770302Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9773111Z STAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9778291Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:36.9783312Z STAGE:2022-09-27 16:02:36 27843:27843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9783764Z STAGE:2022-09-27 16:02:36 27841:27841 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:36.9784351Z STAGE:2022-09-27 16:02:36 27842:27842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:37.3223373Z ok (2.148s)
2022-09-27T16:02:37.3223568Z 
2022-09-27T16:02:37.3224126Z ----------------------------------------------------------------------
2022-09-27T16:02:37.3224391Z Ran 1 test in 2.148s
2022-09-27T16:02:37.3224583Z 
2022-09-27T16:02:37.3224645Z OK
2022-09-27T16:02:37.3224737Z 
2022-09-27T16:02:37.3224830Z Generating XML reports...
2022-09-27T16:02:37.3271194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160235.xml
2022-09-27T16:02:38.6274964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:38.6275600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:38.6283790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:38.6284421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:38.9664047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:38.9673056Z 
2022-09-27T16:02:38.9673153Z Running tests...
2022-09-27T16:02:38.9673729Z ----------------------------------------------------------------------
2022-09-27T16:02:39.2071134Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27906
2022-09-27T16:02:39.2096113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27907
2022-09-27T16:02:39.2123166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27908
2022-09-27T16:02:40.2982812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:40.2983450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:40.3016501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:40.3017166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:40.3257517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:40.3258182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:40.3331141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:40.3331996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:40.3369046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:40.3369819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:40.3592438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:40.3593085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:40.6853748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:40.6955162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:40.6955671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:40.6956324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:40.6956961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:40.6957813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:40.6964300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:40.6964935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:40.6965725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:40.7071447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:40.7173335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:40.7174055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:40.7174974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:40.7175622Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:40.7176188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:40.7178248Z STAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7178921Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7179728Z STAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7185266Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7186013Z STAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7186684Z STAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7192673Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7193859Z STAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7198602Z STAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7202563Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7204343Z STAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7204642Z 
2022-09-27T16:02:40.7208489Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7211840Z STAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7218021Z STAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:40.7223485Z STAGE:2022-09-27 16:02:40 27908:27908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7224342Z STAGE:2022-09-27 16:02:40 27906:27906 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:40 27907:27907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:40.7224714Z 
2022-09-27T16:02:41.1172470Z ok (2.150s)
2022-09-27T16:02:41.1172676Z 
2022-09-27T16:02:41.1173217Z ----------------------------------------------------------------------
2022-09-27T16:02:41.1173573Z Ran 1 test in 2.150s
2022-09-27T16:02:41.1173678Z 
2022-09-27T16:02:41.1173740Z OK
2022-09-27T16:02:41.1173831Z 
2022-09-27T16:02:41.1173926Z Generating XML reports...
2022-09-27T16:02:41.1219267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160238.xml
2022-09-27T16:02:42.4582266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:42.4582642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:42.4589560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:42.4589924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:42.7962765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:42.7972087Z 
2022-09-27T16:02:42.7972195Z Running tests...
2022-09-27T16:02:42.7973293Z ----------------------------------------------------------------------
2022-09-27T16:02:43.0330062Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27971
2022-09-27T16:02:43.0355487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27972
2022-09-27T16:02:43.0382343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27973
2022-09-27T16:02:44.1171725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:44.1172405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:44.1480804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:44.1481405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:44.1739665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:44.1740303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:44.1915302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:44.1915960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:44.2075127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:44.2075546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:44.2253042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:44.2253646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:44.5627525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:44.5727716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:44.5728434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:44.5729131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:44.5729669Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:44.5730192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:44.5836108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:44.5836803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:44.6737864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:44.6847430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:44.6948493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:44.6949156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:44.6949948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:44.6950484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:44.6951016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:44.6954322Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7053606Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7054945Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7060111Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7060806Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7061541Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7068307Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7068890Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7075096Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7079626Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7080415Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7081107Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7085190Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7086242Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7091832Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:44.7096673Z STAGE:2022-09-27 16:02:44 27971:27971 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7097538Z STAGE:2022-09-27 16:02:44 27972:27972 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:44.7097991Z STAGE:2022-09-27 16:02:44 27973:27973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:45.1436065Z ok (2.346s)
2022-09-27T16:02:45.1436270Z 
2022-09-27T16:02:45.1436725Z ----------------------------------------------------------------------
2022-09-27T16:02:45.1437140Z Ran 1 test in 2.346s
2022-09-27T16:02:45.1437308Z 
2022-09-27T16:02:45.1437400Z OK
2022-09-27T16:02:45.1437538Z 
2022-09-27T16:02:45.1437668Z Generating XML reports...
2022-09-27T16:02:45.1484774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160242.xml
2022-09-27T16:02:46.4813264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:46.4813705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:46.4821276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:46.4821636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:46.8187085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:46.8196942Z 
2022-09-27T16:02:46.8197354Z Running tests...
2022-09-27T16:02:46.8197726Z ----------------------------------------------------------------------
2022-09-27T16:02:47.0555594Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28036
2022-09-27T16:02:47.0582538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28037
2022-09-27T16:02:47.0609674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28038
2022-09-27T16:02:48.1231109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:48.1231550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:48.1554158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:48.1554669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:48.1555464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:48.1555807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:48.1602022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:48.1602361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:48.1914562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:48.1915337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:48.1964955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:48.1965624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:48.5461672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:48.5562142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:48.5562765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:48.5563640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:48.5564221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:48.5565054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:48.5570843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:48.5572355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:48.5573700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:48.5780214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:48.5882025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:48.5882585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:48.5883248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:48.5883793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:48.5884316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:48.5887645Z STAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5887968Z 
2022-09-27T16:02:48.5888226Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5893334Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5894096Z STAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5894555Z STAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5901203Z STAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5901749Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5908111Z STAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5912704Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5913424Z STAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5914137Z STAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5919607Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5920075Z STAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5925841Z STAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:48.5930089Z STAGE:2022-09-27 16:02:48 28038:28038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5930795Z STAGE:2022-09-27 16:02:48 28036:28036 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:48 28037:28037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:48.5931130Z 
2022-09-27T16:02:48.9658720Z ok (2.146s)
2022-09-27T16:02:48.9658975Z 
2022-09-27T16:02:48.9659469Z ----------------------------------------------------------------------
2022-09-27T16:02:48.9659910Z Ran 1 test in 2.146s
2022-09-27T16:02:48.9660100Z 
2022-09-27T16:02:48.9660214Z OK
2022-09-27T16:02:48.9660379Z 
2022-09-27T16:02:48.9660545Z Generating XML reports...
2022-09-27T16:02:48.9705297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160246.xml
2022-09-27T16:02:50.2917388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:50.2918015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:50.2925249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:50.2925663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:50.6475089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:50.6485051Z 
2022-09-27T16:02:50.6485178Z Running tests...
2022-09-27T16:02:50.6485793Z ----------------------------------------------------------------------
2022-09-27T16:02:50.8919669Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28101
2022-09-27T16:02:50.8947823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28102
2022-09-27T16:02:50.8975048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28103
2022-09-27T16:02:51.9982515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:51.9983148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:52.0105052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:52.0105675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:52.0322820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:52.0323429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:52.0380860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:52.0381237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:52.0447923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:52.0448339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:52.0699717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:52.0700096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:52.3994600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:52.3995319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:52.3995713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:52.3996335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:52.3996878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:52.3997407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:52.4100461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:52.5003279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:52.5004015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:52.5005317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:52.5110724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:52.5117082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:52.5118107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:52.5121803Z STAGE:2022-09-27 16:02:52 28102:28102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:52.5209433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:52.5212557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:52.5216589Z STAGE:2022-09-27 16:02:52 28103:28103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:52.5222956Z STAGE:2022-09-27 16:02:52 28103:28103 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:52 28102:28102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:52.5223275Z 
2022-09-27T16:02:52.5230916Z STAGE:2022-09-27 16:02:52 28103:28103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:52.5236312Z STAGE:2022-09-27 16:02:52 28102:28102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:52.5241730Z STAGE:2022-09-27 16:02:52 28102:28102 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:52 28103:28103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:52.5242045Z 
2022-09-27T16:02:52.9025427Z ok (2.254s)
2022-09-27T16:02:52.9025604Z 
2022-09-27T16:02:52.9025917Z ----------------------------------------------------------------------
2022-09-27T16:02:52.9026157Z Ran 1 test in 2.254s
2022-09-27T16:02:52.9026275Z 
2022-09-27T16:02:52.9026338Z OK
2022-09-27T16:02:52.9026431Z 
2022-09-27T16:02:52.9026537Z Generating XML reports...
2022-09-27T16:02:52.9070525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160250.xml
2022-09-27T16:02:54.2516298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:54.2516680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:54.2523238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:54.2523791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:54.5927833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:54.5937210Z 
2022-09-27T16:02:54.5937342Z Running tests...
2022-09-27T16:02:54.5937952Z ----------------------------------------------------------------------
2022-09-27T16:02:54.8304592Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28162
2022-09-27T16:02:54.8330633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28163
2022-09-27T16:02:54.8357831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28164
2022-09-27T16:02:55.8867683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:55.8868068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:55.9185429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:55.9185893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:55.9247025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:55.9257392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:55.9258220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:55.9258935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:55.9600105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:55.9600751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:55.9607794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:55.9608438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:56.2986628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:56.3072379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:02:56.3073036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:56.3073661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:56.3074186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:56.3087711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:02:56.3179783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:02:56.3180413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:56.3386041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:02:56.3386447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:56.4096547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:56.4099502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:56.4100113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:56.4195226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:56.4196043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:02:56.4199361Z STAGE:2022-09-27 16:02:56 28163:28163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:56.4299455Z STAGE:2022-09-27 16:02:56 28164:28164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:56.4305996Z STAGE:2022-09-27 16:02:56 28164:28164 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:56.4306736Z STAGE:2022-09-27 16:02:56 28163:28163 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:56.4312814Z STAGE:2022-09-27 16:02:56 28164:28164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:56.4319653Z STAGE:2022-09-27 16:02:56 28163:28163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:56.4324701Z STAGE:2022-09-27 16:02:56 28163:28163 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:02:56 28164:28164 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:56.4325175Z 
2022-09-27T16:02:56.8408216Z ok (2.247s)
2022-09-27T16:02:56.8408495Z 
2022-09-27T16:02:56.8409018Z ----------------------------------------------------------------------
2022-09-27T16:02:56.8409398Z Ran 1 test in 2.247s
2022-09-27T16:02:56.8409551Z 
2022-09-27T16:02:56.8409858Z OK
2022-09-27T16:02:56.8409997Z 
2022-09-27T16:02:56.8410134Z Generating XML reports...
2022-09-27T16:02:56.8455358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160254.xml
2022-09-27T16:02:58.1697334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:58.1697748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:58.1705107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:58.1705478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:58.5091971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:58.5101263Z 
2022-09-27T16:02:58.5101365Z Running tests...
2022-09-27T16:02:58.5101933Z ----------------------------------------------------------------------
2022-09-27T16:02:58.7491816Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28223
2022-09-27T16:02:58.7518520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28224
2022-09-27T16:02:58.7545645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28225
2022-09-27T16:02:59.8659024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:59.8659402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:59.8685123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:59.8685589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:59.9004173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:59.9004906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:59.9028839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:59.9029414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:59.9034320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:59.9034912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:59.9349980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:59.9350618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:00.2701179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:00.2786743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:00.2787203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:00.2787861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:00.2788393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:00.2802404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:00.2894173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:00.2894845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:00.3099065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:00.3099469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:03:00.3810748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:00.3813449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:00.3814090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:00.3907148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:00.3908012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:00.3911177Z STAGE:2022-09-27 16:03:00 28224:28224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:00.3911629Z STAGE:2022-09-27 16:03:00 28225:28225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:00.3917239Z STAGE:2022-09-27 16:03:00 28225:28225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:00.3917986Z STAGE:2022-09-27 16:03:00 28224:28224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:00.3924565Z STAGE:2022-09-27 16:03:00 28225:28225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:00.3931112Z STAGE:2022-09-27 16:03:00 28224:28224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:00.3935870Z STAGE:2022-09-27 16:03:00 28225:28225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:00.3936617Z STAGE:2022-09-27 16:03:00 28224:28224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:00.8600141Z ok (2.349s)
2022-09-27T16:03:00.8600363Z 
2022-09-27T16:03:00.8600714Z ----------------------------------------------------------------------
2022-09-27T16:03:00.8601012Z Ran 1 test in 2.350s
2022-09-27T16:03:00.8601151Z 
2022-09-27T16:03:00.8601202Z OK
2022-09-27T16:03:00.8601295Z 
2022-09-27T16:03:00.8601399Z Generating XML reports...
2022-09-27T16:03:00.8646170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160258.xml
2022-09-27T16:03:02.1945822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:02.1946507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:02.1954934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:02.1955573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:02.5365588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:02.5375004Z 
2022-09-27T16:03:02.5375524Z Running tests...
2022-09-27T16:03:02.5375918Z ----------------------------------------------------------------------
2022-09-27T16:03:02.7769792Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28284
2022-09-27T16:03:02.7796758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28285
2022-09-27T16:03:02.7823983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28286
2022-09-27T16:03:03.8250933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.8251371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.8571200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.8571604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:03.8730872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.8731234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.9047322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.9047723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.9082386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.9082929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:03.9369405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.9370040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:04.2562489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:04.2642632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:04.2643048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:04.2643659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:04.2644198Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:04.2663231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:04.2749972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:04.2750772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:04.2955987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:04.2956554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:03:04.3671824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:04.3674481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:04.3675143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:04.3764636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:04.3765482Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:04.3768499Z STAGE:2022-09-27 16:03:04 28285:28285 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:03:04 28286:28286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:04.3768815Z 
2022-09-27T16:03:04.3773859Z STAGE:2022-09-27 16:03:04 28286:28286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:04.3774303Z STAGE:2022-09-27 16:03:04 28285:28285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:04.3781046Z STAGE:2022-09-27 16:03:04 28286:28286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:04.3786753Z STAGE:2022-09-27 16:03:04 28285:28285 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:04.3791314Z STAGE:2022-09-27 16:03:04 28285:28285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:04.3791750Z STAGE:2022-09-27 16:03:04 28286:28286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:04.7874312Z ok (2.249s)
2022-09-27T16:03:04.7874817Z 
2022-09-27T16:03:04.7875277Z ----------------------------------------------------------------------
2022-09-27T16:03:04.7875680Z Ran 1 test in 2.250s
2022-09-27T16:03:04.7875969Z 
2022-09-27T16:03:04.7876064Z OK
2022-09-27T16:03:04.7876217Z 
2022-09-27T16:03:04.7876372Z Generating XML reports...
2022-09-27T16:03:04.7922319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160302.xml
2022-09-27T16:03:06.1493780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:06.1494480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:06.1503132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:06.1503802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:06.5053122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:06.5064350Z 
2022-09-27T16:03:06.7453628Z Running tests...
2022-09-27T16:03:06.7454311Z ----------------------------------------------------------------------
2022-09-27T16:03:06.7455020Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28345
2022-09-27T16:03:06.7481423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28346
2022-09-27T16:03:06.7509819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28347
2022-09-27T16:03:07.8126935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:07.8127566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:07.8235220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:07.8235865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:07.8408638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:07.8409353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:07.8466636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:07.8467183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:07.8586426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:07.8586988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:07.8741542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:07.8742185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:08.1965357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:08.1965880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:08.1966475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:08.1967326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:08.1968249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:08.1968833Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:08.2071640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:08.2075928Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.2975917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:08.2976471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:08.2981482Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.2981796Z 
2022-09-27T16:03:08.2986568Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.2987122Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.2998363Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.2999003Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.3001374Z STAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3015231Z STAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.3020313Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3021194Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3021760Z STAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3029989Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.3030678Z STAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.3033121Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:08.3038774Z STAGE:2022-09-27 16:03:08 28346:28346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3039486Z STAGE:2022-09-27 16:03:08 28345:28345 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.3039928Z STAGE:2022-09-27 16:03:08 28347:28347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:08.6558702Z ok (2.149s)
2022-09-27T16:03:08.6559036Z 
2022-09-27T16:03:08.6559821Z ----------------------------------------------------------------------
2022-09-27T16:03:08.6560243Z Ran 1 test in 2.149s
2022-09-27T16:03:08.6560406Z 
2022-09-27T16:03:08.6560494Z OK
2022-09-27T16:03:08.6560601Z 
2022-09-27T16:03:08.6560697Z Generating XML reports...
2022-09-27T16:03:08.6605768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160306.xml
2022-09-27T16:03:09.9684793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:09.9685196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:09.9692936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:09.9693329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:10.3115994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:10.3125441Z 
2022-09-27T16:03:10.3125849Z Running tests...
2022-09-27T16:03:10.3126257Z ----------------------------------------------------------------------
2022-09-27T16:03:10.5491047Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28401
2022-09-27T16:03:10.5517481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28402
2022-09-27T16:03:10.5544596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28403
2022-09-27T16:03:11.6330380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:11.6331333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:11.6567741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:11.6568354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:11.6658266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:11.6658660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:11.6686920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:11.6687522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:11.6916809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:11.6917242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:11.7033896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:11.7034425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:12.0391700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:12.0491880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:12.0492533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:12.0493610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:12.0494146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:12.0494669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:12.0599118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:12.0602514Z STAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1503215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:12.1508017Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1517183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:12.1521870Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1534566Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1536304Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1536786Z 
2022-09-27T16:03:12.1543302Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1550105Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1556532Z STAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1562659Z STAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1563515Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1564027Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1569449Z STAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1570398Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1576114Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:12.1580745Z STAGE:2022-09-27 16:03:12 28402:28402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1581966Z STAGE:2022-09-27 16:03:12 28401:28401 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:12 28403:28403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:12.1582314Z 
2022-09-27T16:03:12.5598924Z ok (2.247s)
2022-09-27T16:03:12.5599178Z 
2022-09-27T16:03:12.5599648Z ----------------------------------------------------------------------
2022-09-27T16:03:12.5599931Z Ran 1 test in 2.247s
2022-09-27T16:03:12.5600047Z 
2022-09-27T16:03:12.5600106Z OK
2022-09-27T16:03:12.5600197Z 
2022-09-27T16:03:12.5600290Z Generating XML reports...
2022-09-27T16:03:12.5643924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160310.xml
2022-09-27T16:03:13.8774347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:13.8774725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:13.8782609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:13.8783121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:14.2173927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:14.2183162Z 
2022-09-27T16:03:14.2183618Z Running tests...
2022-09-27T16:03:14.2184420Z ----------------------------------------------------------------------
2022-09-27T16:03:14.2202328Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T16:03:14.2202770Z 
2022-09-27T16:03:14.2203166Z ----------------------------------------------------------------------
2022-09-27T16:03:14.2203568Z Ran 1 test in 0.002s
2022-09-27T16:03:14.2203754Z 
2022-09-27T16:03:14.2203877Z OK (skipped=1)
2022-09-27T16:03:14.2204066Z 
2022-09-27T16:03:14.2204199Z Generating XML reports...
2022-09-27T16:03:14.2234782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160314.xml
2022-09-27T16:03:15.4121405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:15.4121786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:15.4129065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:15.4129432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:15.7513557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:15.7523325Z 
2022-09-27T16:03:15.7523435Z Running tests...
2022-09-27T16:03:15.7523872Z ----------------------------------------------------------------------
2022-09-27T16:03:15.9866094Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28467
2022-09-27T16:03:15.9892503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28468
2022-09-27T16:03:15.9919752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28469
2022-09-27T16:03:17.0390376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:17.0391356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:17.0706598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:17.0707181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:17.0764961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:17.0765581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:17.1089663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:17.1090057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:17.1170443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:17.1170802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:17.1491559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:17.1492017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:17.4992272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:17.5093965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:17.5094422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:17.5095073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:17.5095612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:17.5096128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:17.5103455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:17.5107683Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6103811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:17.6104434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:17.6107673Z STAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6109309Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6115028Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6115852Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6116302Z STAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6123098Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6125119Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6130390Z STAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6135372Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6136348Z STAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6137101Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6141373Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6142467Z STAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6148387Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:17.6154165Z STAGE:2022-09-27 16:03:17 28468:28468 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:17 28467:28467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.6154488Z 
2022-09-27T16:03:17.6154754Z STAGE:2022-09-27 16:03:17 28469:28469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:17.9972057Z ok (2.244s)
2022-09-27T16:03:17.9973076Z 
2022-09-27T16:03:17.9973392Z ----------------------------------------------------------------------
2022-09-27T16:03:17.9973647Z Ran 1 test in 2.245s
2022-09-27T16:03:17.9973774Z 
2022-09-27T16:03:17.9973821Z OK
2022-09-27T16:03:17.9973916Z 
2022-09-27T16:03:17.9974010Z Generating XML reports...
2022-09-27T16:03:18.0019391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160315.xml
2022-09-27T16:03:19.3084743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:19.3085198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:19.3093109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:19.3093477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:19.6495690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:19.6505057Z 
2022-09-27T16:03:19.6505199Z Running tests...
2022-09-27T16:03:19.6505826Z ----------------------------------------------------------------------
2022-09-27T16:03:19.6531244Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2022-09-27T16:03:19.6531623Z 
2022-09-27T16:03:19.6532012Z ----------------------------------------------------------------------
2022-09-27T16:03:19.6532460Z Ran 1 test in 0.003s
2022-09-27T16:03:19.6532634Z 
2022-09-27T16:03:19.6533437Z OK (skipped=1)
2022-09-27T16:03:19.6533570Z 
2022-09-27T16:03:19.6533693Z Generating XML reports...
2022-09-27T16:03:19.6562725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160319.xml
2022-09-27T16:03:20.8401136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.8401516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.8408477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.8409036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:21.1781568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:21.1790879Z 
2022-09-27T16:03:21.1791016Z Running tests...
2022-09-27T16:03:21.1791623Z ----------------------------------------------------------------------
2022-09-27T16:03:21.4170302Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28533
2022-09-27T16:03:21.4203367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28534
2022-09-27T16:03:21.4237658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28535
2022-09-27T16:03:22.5265107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:22.5265501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:22.5431750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:22.5432138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:22.5558645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:22.5559176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:22.5602094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:22.5602468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:22.5786554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:22.5787175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:22.5899466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:22.5900121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:22.9143148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:22.9244130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:22.9244773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:22.9245715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:22.9246305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:22.9247042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:22.9252929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:22.9253290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:22.9255368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:22.9259017Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9260284Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9260987Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9265800Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9266466Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9266921Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9273144Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9275518Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9282019Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9287047Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9287802Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9288401Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9295598Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9296537Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9299808Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:22.9304310Z STAGE:2022-09-27 16:03:22 28533:28533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9305135Z STAGE:2022-09-27 16:03:22 28534:28534 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:22.9305931Z STAGE:2022-09-27 16:03:22 28535:28535 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:23.3286726Z ok (2.149s)
2022-09-27T16:03:23.3286948Z 
2022-09-27T16:03:23.3287293Z ----------------------------------------------------------------------
2022-09-27T16:03:23.3287546Z Ran 1 test in 2.149s
2022-09-27T16:03:23.3287652Z 
2022-09-27T16:03:23.3287716Z OK
2022-09-27T16:03:23.3287816Z 
2022-09-27T16:03:23.3287907Z Generating XML reports...
2022-09-27T16:03:23.3333062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160321.xml
2022-09-27T16:03:24.6479839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:24.6480230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:24.6488087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:24.6488659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:24.9852122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:24.9861424Z 
2022-09-27T16:03:24.9861848Z Running tests...
2022-09-27T16:03:24.9862242Z ----------------------------------------------------------------------
2022-09-27T16:03:24.9882401Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:03:24.9882671Z 
2022-09-27T16:03:24.9882924Z ----------------------------------------------------------------------
2022-09-27T16:03:24.9883240Z Ran 1 test in 0.002s
2022-09-27T16:03:24.9883343Z 
2022-09-27T16:03:24.9883417Z OK (skipped=1)
2022-09-27T16:03:24.9883528Z 
2022-09-27T16:03:24.9883614Z Generating XML reports...
2022-09-27T16:03:24.9913187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160324.xml
2022-09-27T16:03:26.1886153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:26.1886816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:26.1895119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:26.1895779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:26.5256292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:26.5265957Z 
2022-09-27T16:03:26.5266372Z Running tests...
2022-09-27T16:03:26.5266776Z ----------------------------------------------------------------------
2022-09-27T16:03:26.5285665Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:03:26.5285959Z 
2022-09-27T16:03:26.5286289Z ----------------------------------------------------------------------
2022-09-27T16:03:26.5286546Z Ran 1 test in 0.002s
2022-09-27T16:03:26.5286712Z 
2022-09-27T16:03:26.5286836Z OK (skipped=1)
2022-09-27T16:03:26.5286970Z 
2022-09-27T16:03:26.5287289Z Generating XML reports...
2022-09-27T16:03:26.5316287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160326.xml
2022-09-27T16:03:27.7254072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:27.7254469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:27.7262092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:27.7262452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:28.0653888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:28.0663209Z 
2022-09-27T16:03:28.0663347Z Running tests...
2022-09-27T16:03:28.0663748Z ----------------------------------------------------------------------
2022-09-27T16:03:28.3026594Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28609
2022-09-27T16:03:28.3053333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28610
2022-09-27T16:03:28.3080200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28611
2022-09-27T16:03:29.3355845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.3356233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.3661842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.3662529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.3675510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.3676236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:29.3775291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.3775652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.4006334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.4006792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:29.4117107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.4117726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:29.7501483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:29.7502268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:29.7503022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:29.7503442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:29.7504047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:29.7504626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:29.7607325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:29.7611648Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8511995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:29.8513203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:29.8516921Z STAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8517794Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8525764Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8526537Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8526984Z STAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8535771Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8539557Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8547345Z STAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8554778Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8555249Z STAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8555690Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8561566Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8565963Z STAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8573192Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:29.8581382Z STAGE:2022-09-27 16:03:29 28610:28610 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:29 28609:28609 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:29.8581881Z 
2022-09-27T16:03:29.8582318Z STAGE:2022-09-27 16:03:29 28611:28611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:30.2129264Z ok (2.146s)
2022-09-27T16:03:30.2129488Z 
2022-09-27T16:03:30.2129991Z ----------------------------------------------------------------------
2022-09-27T16:03:30.2130387Z Ran 1 test in 2.147s
2022-09-27T16:03:30.2130503Z 
2022-09-27T16:03:30.2130552Z OK
2022-09-27T16:03:30.2130645Z 
2022-09-27T16:03:30.2130741Z Generating XML reports...
2022-09-27T16:03:30.2175193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160328.xml
2022-09-27T16:03:31.5302473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:31.5302859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:31.5310514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:31.5310888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:31.8701588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:31.8711059Z 
2022-09-27T16:03:31.8711515Z Running tests...
2022-09-27T16:03:31.8711926Z ----------------------------------------------------------------------
2022-09-27T16:03:32.1064412Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28665
2022-09-27T16:03:32.1091840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28666
2022-09-27T16:03:32.1118155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28667
2022-09-27T16:03:33.1972258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:33.1973180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:33.2247209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:33.2248151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:33.2272958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:33.2273356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:33.2302012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:33.2302558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:33.2605779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:33.2606413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:33.2631147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:33.2631577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:33.6054027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:33.6154675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:33.6155364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:33.6156618Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:33.6157256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:33.6157788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:33.6261208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:33.6266181Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7166826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:33.7167213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:33.7172597Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7173068Z STAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7178348Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7178666Z 
2022-09-27T16:03:33.7178959Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7188484Z STAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7189105Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7189631Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7192849Z STAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7196173Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7198284Z STAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7202248Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7204195Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7210664Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:33.7214164Z STAGE:2022-09-27 16:03:33 28667:28667 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7215003Z STAGE:2022-09-27 16:03:33 28665:28665 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:33.7215448Z STAGE:2022-09-27 16:03:33 28666:28666 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:34.1169940Z ok (2.245s)
2022-09-27T16:03:34.1170161Z 
2022-09-27T16:03:34.1170611Z ----------------------------------------------------------------------
2022-09-27T16:03:34.1171033Z Ran 1 test in 2.246s
2022-09-27T16:03:34.1171209Z 
2022-09-27T16:03:34.1171302Z OK
2022-09-27T16:03:34.1171459Z 
2022-09-27T16:03:34.1171595Z Generating XML reports...
2022-09-27T16:03:34.1217597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160331.xml
2022-09-27T16:03:35.4213872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:35.4214297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:35.4221891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:35.4222263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:35.7616136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:35.7625749Z 
2022-09-27T16:03:35.7626254Z Running tests...
2022-09-27T16:03:35.7626673Z ----------------------------------------------------------------------
2022-09-27T16:03:36.0014092Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28721
2022-09-27T16:03:36.0040435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28722
2022-09-27T16:03:36.0067184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28723
2022-09-27T16:03:37.0939559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:37.0940176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:37.1255757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:37.1256159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:37.1276253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:37.1276729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:37.1279348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:37.1279973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:37.1630477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:37.1630862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:37.1631324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:37.1631689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:37.4953854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:37.5054909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:37.5055311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:37.5056022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:37.5056556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:37.5057089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:37.5061486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:37.5063151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:37.5065432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:37.8118238Z ok (2.049s)
2022-09-27T16:03:37.8118501Z 
2022-09-27T16:03:37.8118957Z ----------------------------------------------------------------------
2022-09-27T16:03:37.8119237Z Ran 1 test in 2.049s
2022-09-27T16:03:37.8119356Z 
2022-09-27T16:03:37.8119405Z OK
2022-09-27T16:03:37.8119499Z 
2022-09-27T16:03:37.8119593Z Generating XML reports...
2022-09-27T16:03:37.8164669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160335.xml
2022-09-27T16:03:39.1120225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:39.1120596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:39.1128381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:39.1128744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:39.4511392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:39.4520367Z 
2022-09-27T16:03:39.4520501Z Running tests...
2022-09-27T16:03:39.4521142Z ----------------------------------------------------------------------
2022-09-27T16:03:39.6861075Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28774
2022-09-27T16:03:39.6887243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28775
2022-09-27T16:03:39.6914370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28776
2022-09-27T16:03:40.7759457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:40.7760115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:40.8030107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:40.8030740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:40.8086952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:40.8087578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:40.8360878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:40.8361545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:40.8543342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:40.8543703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:40.8871696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:40.8872349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:41.2355840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:41.2356631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:41.2357003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:41.2357623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:41.2358160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:41.2358687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:41.2462154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:41.2462546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:41.2467676Z STAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.2468524Z STAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3366210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:41.3371042Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3377955Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3378710Z 
2022-09-27T16:03:41.3378719Z 
2022-09-27T16:03:41.3389124Z STAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3390188Z STAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3390936Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3395715Z STAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3396257Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3396684Z STAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3402174Z STAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3404450Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3406297Z STAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:41.3408596Z STAGE:2022-09-27 16:03:41 28774:28774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3410966Z STAGE:2022-09-27 16:03:41 28775:28775 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:41 28776:28776 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:41.3411404Z 
2022-09-27T16:03:41.6967346Z ok (2.244s)
2022-09-27T16:03:41.6967577Z 
2022-09-27T16:03:41.6967950Z ----------------------------------------------------------------------
2022-09-27T16:03:41.6968231Z Ran 1 test in 2.245s
2022-09-27T16:03:41.6968351Z 
2022-09-27T16:03:41.6968398Z OK
2022-09-27T16:03:41.6968490Z 
2022-09-27T16:03:41.6968583Z Generating XML reports...
2022-09-27T16:03:41.7014380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160339.xml
2022-09-27T16:03:42.9972204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:42.9972812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:42.9980691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:42.9981195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:43.3384766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:43.3393598Z 
2022-09-27T16:03:43.3393758Z Running tests...
2022-09-27T16:03:43.3394186Z ----------------------------------------------------------------------
2022-09-27T16:03:43.3411773Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-09-27T16:03:43.3412020Z 
2022-09-27T16:03:43.3412302Z ----------------------------------------------------------------------
2022-09-27T16:03:43.3412551Z Ran 1 test in 0.002s
2022-09-27T16:03:43.3412666Z 
2022-09-27T16:03:43.3412740Z OK (skipped=1)
2022-09-27T16:03:43.3412855Z 
2022-09-27T16:03:43.3412928Z Generating XML reports...
2022-09-27T16:03:43.3442028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160343.xml
2022-09-27T16:03:44.5303548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.5304170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.5311554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.5312072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.8677591Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:44.8686363Z 
2022-09-27T16:03:44.8686519Z Running tests...
2022-09-27T16:03:44.8687168Z ----------------------------------------------------------------------
2022-09-27T16:03:44.8704679Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:03:44.8705043Z 
2022-09-27T16:03:44.8705444Z ----------------------------------------------------------------------
2022-09-27T16:03:44.8705841Z Ran 1 test in 0.002s
2022-09-27T16:03:44.8705959Z 
2022-09-27T16:03:44.8706033Z OK (skipped=1)
2022-09-27T16:03:44.8706142Z 
2022-09-27T16:03:44.8706215Z Generating XML reports...
2022-09-27T16:03:44.8735579Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160344.xml
2022-09-27T16:03:46.0642197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:46.0650313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:46.0650799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:46.0651166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:46.4038700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:46.4049022Z 
2022-09-27T16:03:46.4049355Z Running tests...
2022-09-27T16:03:46.4049946Z ----------------------------------------------------------------------
2022-09-27T16:03:46.6406935Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28850
2022-09-27T16:03:46.6434430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28851
2022-09-27T16:03:46.6461148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28852
2022-09-27T16:03:47.6864908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:47.6865675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:47.7016755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:47.7017308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:47.7017936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:47.7018425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:47.7214884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:47.7215484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:47.7373794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:47.7374473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:47.7380261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:47.7380940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:48.0823780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:48.0824781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:48.0825176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:48.0825808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:48.0826348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:48.0826879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:48.0929851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:48.1834587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:48.1834974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:48.2040467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:48.2142032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:48.2142763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:03:48.2143493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:48.2144192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:48.2144805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:48.2147959Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2148252Z 
2022-09-27T16:03:48.2153219Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2163061Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2247675Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2252213Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2252720Z STAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2265759Z STAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2266424Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2271123Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2271932Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2272226Z 
2022-09-27T16:03:48.2277104Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2278382Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:48.2278764Z 
2022-09-27T16:03:48.2282085Z STAGE:2022-09-27 16:03:48 28851:28851 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2282684Z STAGE:2022-09-27 16:03:48 28850:28850 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.2283217Z STAGE:2022-09-27 16:03:48 28852:28852 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:48.6514461Z ok (2.246s)
2022-09-27T16:03:48.6514706Z 
2022-09-27T16:03:48.6515167Z ----------------------------------------------------------------------
2022-09-27T16:03:48.6515529Z Ran 1 test in 2.246s
2022-09-27T16:03:48.6515705Z 
2022-09-27T16:03:48.6515792Z OK
2022-09-27T16:03:48.6515938Z 
2022-09-27T16:03:48.6516101Z Generating XML reports...
2022-09-27T16:03:48.6562858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160346.xml
2022-09-27T16:03:49.9614167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:49.9614834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:49.9623453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:49.9624321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:50.3016345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:50.3026429Z 
2022-09-27T16:03:50.3026963Z Running tests...
2022-09-27T16:03:50.3027390Z ----------------------------------------------------------------------
2022-09-27T16:03:50.5424028Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28915
2022-09-27T16:03:50.5450835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28916
2022-09-27T16:03:50.5477732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28917
2022-09-27T16:03:51.6359009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:51.6359677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:51.6673425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:51.6674044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:51.6957856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:51.6958466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:51.7265109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:51.7265695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:51.7272980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:51.7273342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:51.7592360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:51.7592991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:52.0960821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:52.0961521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:52.0962162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:52.0962557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:52.0963067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:52.0963584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:52.1067148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:52.1067567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:52.1273265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:52.1273671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:03:52.1970149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:52.1972016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:52.1972611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:52.1980403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:52.1980990Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-09-27T16:03:52.1985053Z STAGE:2022-09-27 16:03:52 28917:28917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:52.1985842Z STAGE:2022-09-27 16:03:52 28916:28916 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:52.1990972Z STAGE:2022-09-27 16:03:52 28916:28916 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:52.1991516Z STAGE:2022-09-27 16:03:52 28917:28917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:52.2001307Z STAGE:2022-09-27 16:03:52 28916:28916 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:03:52 28917:28917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:03:52.2001715Z 
2022-09-27T16:03:52.2005459Z STAGE:2022-09-27 16:03:52 28917:28917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:52.2006112Z STAGE:2022-09-27 16:03:52 28916:28916 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:03:52.6532983Z ok (2.350s)
2022-09-27T16:03:52.6533223Z 
2022-09-27T16:03:52.6533754Z ----------------------------------------------------------------------
2022-09-27T16:03:52.6534201Z Ran 1 test in 2.351s
2022-09-27T16:03:52.6534326Z 
2022-09-27T16:03:52.6534605Z OK
2022-09-27T16:03:52.6534702Z 
2022-09-27T16:03:52.6534798Z Generating XML reports...
2022-09-27T16:03:52.6579272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160350.xml
2022-09-27T16:03:53.9687095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:53.9687497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:53.9694973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:53.9695338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:54.3085763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:54.3095074Z 
2022-09-27T16:03:54.3095206Z Running tests...
2022-09-27T16:03:54.3095821Z ----------------------------------------------------------------------
2022-09-27T16:03:54.5484957Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28976
2022-09-27T16:03:54.5511785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28977
2022-09-27T16:03:54.5539054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28978
2022-09-27T16:03:55.6147706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:55.6148118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:55.6405428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:55.6405781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:55.6407671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:55.6408023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:55.6468830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:55.6469207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:55.6756849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:55.6757225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:55.6763892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:55.6764450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:56.0145161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:56.0145787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:56.0146221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:56.0147258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:56.0147834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:56.0148356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:56.0250590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:56.1154963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:56.1155712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:03:56.4592684Z ok (2.149s)
2022-09-27T16:03:56.4592878Z 
2022-09-27T16:03:56.4593333Z ----------------------------------------------------------------------
2022-09-27T16:03:56.4594108Z Ran 1 test in 2.150s
2022-09-27T16:03:56.4594259Z 
2022-09-27T16:03:56.4594307Z OK
2022-09-27T16:03:56.4594405Z 
2022-09-27T16:03:56.4594500Z Generating XML reports...
2022-09-27T16:03:56.4639956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160354.xml
2022-09-27T16:03:57.7721706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:57.7722111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:57.7729894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:57.7730272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:58.1124924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:58.1134162Z 
2022-09-27T16:03:58.1134555Z Running tests...
2022-09-27T16:03:58.1134968Z ----------------------------------------------------------------------
2022-09-27T16:03:58.3503026Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29029
2022-09-27T16:03:58.3530058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29030
2022-09-27T16:03:58.3556795Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29031
2022-09-27T16:03:59.4497768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.4498435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.4644740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.4645277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.4655795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.4656233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.4842784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.4843242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.5005392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.5005946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.5014687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.5015191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.8450170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:59.8551041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:59.8551670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:03:59.8552577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:59.8553492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:59.8554126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:03:59.8657732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:59.9561243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:59.9561949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:00.2608588Z ok (2.147s)
2022-09-27T16:04:00.2608875Z 
2022-09-27T16:04:00.2609307Z ----------------------------------------------------------------------
2022-09-27T16:04:00.2609593Z Ran 1 test in 2.147s
2022-09-27T16:04:00.2609709Z 
2022-09-27T16:04:00.2609782Z OK
2022-09-27T16:04:00.2609862Z 
2022-09-27T16:04:00.2609979Z Generating XML reports...
2022-09-27T16:04:00.2654764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160358.xml
2022-09-27T16:04:01.5686187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:01.5686574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:01.5693824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:01.5694229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:01.9060542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:01.9070022Z 
2022-09-27T16:04:01.9070111Z Running tests...
2022-09-27T16:04:01.9070598Z ----------------------------------------------------------------------
2022-09-27T16:04:02.1436867Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29082
2022-09-27T16:04:02.1464127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29083
2022-09-27T16:04:02.1491395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29084
2022-09-27T16:04:03.2264300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:03.2265025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:03.2459659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:03.2460327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:03.2600071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:03.2600618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:03.2617205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:03.2617735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:03.2817070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:03.2817470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:03.2957305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:03.2957751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:03.6474025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:03.6575296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:03.6575917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:03.6576610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:03.6577617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:03.6578348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:03.6681858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:03.7585903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:03.7586612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:04.0539431Z ok (2.146s)
2022-09-27T16:04:04.0539663Z 
2022-09-27T16:04:04.0540118Z ----------------------------------------------------------------------
2022-09-27T16:04:04.0540504Z Ran 1 test in 2.147s
2022-09-27T16:04:04.0540682Z 
2022-09-27T16:04:04.0540769Z OK
2022-09-27T16:04:04.0540914Z 
2022-09-27T16:04:04.0541058Z Generating XML reports...
2022-09-27T16:04:04.0587908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160401.xml
2022-09-27T16:04:05.3787360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:05.3787809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:05.3795639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:05.3796002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:05.7166562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:05.7176915Z 
2022-09-27T16:04:05.7177180Z Running tests...
2022-09-27T16:04:05.7177770Z ----------------------------------------------------------------------
2022-09-27T16:04:05.9542930Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29135
2022-09-27T16:04:05.9569287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29136
2022-09-27T16:04:05.9596097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29137
2022-09-27T16:04:06.9869130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:06.9869798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:07.0186581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:07.0186994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:07.0252221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:07.0252882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:07.0460550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:07.0461107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:07.0594338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:07.0594704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:07.0792316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:07.0792975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:07.4084674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:07.4185733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:07.4186323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:07.4187012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:07.4187638Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:07.4188166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:07.4292685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:07.4296152Z STAGE:2022-09-27 16:04:07 29137:29137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:07.5197045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:07.5198071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:07.5201028Z STAGE:2022-09-27 16:04:07 29135:29135 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:07.5202015Z STAGE:2022-09-27 16:04:07 29136:29136 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:07.5225556Z STAGE:2022-09-27 16:04:07 29135:29135 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:07.5226547Z STAGE:2022-09-27 16:04:07 29136:29136 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:07.5227318Z STAGE:2022-09-27 16:04:07 29137:29137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:07.8644748Z ok (2.146s)
2022-09-27T16:04:07.8644967Z 
2022-09-27T16:04:07.8645465Z ----------------------------------------------------------------------
2022-09-27T16:04:07.8645904Z Ran 1 test in 2.147s
2022-09-27T16:04:07.8646063Z 
2022-09-27T16:04:07.8646128Z OK
2022-09-27T16:04:07.8646220Z 
2022-09-27T16:04:07.8646314Z Generating XML reports...
2022-09-27T16:04:07.8691713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160405.xml
2022-09-27T16:04:09.1961247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:09.1961642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:09.1969167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:09.1969708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:09.5525129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:09.5534751Z 
2022-09-27T16:04:09.5534853Z Running tests...
2022-09-27T16:04:09.5535340Z ----------------------------------------------------------------------
2022-09-27T16:04:09.7926889Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29191
2022-09-27T16:04:09.7952989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29192
2022-09-27T16:04:09.7980785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29193
2022-09-27T16:04:10.8580730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:10.8581315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:10.8726540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:10.8727208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:10.8913594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:10.8914254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:10.9071863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:10.9072274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:10.9191273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:10.9191721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:10.9516101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:10.9516478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:11.2770334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:11.2770965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:11.2771592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:11.2772575Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:11.2773485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:11.2774017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:11.2876895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:11.2881405Z STAGE:2022-09-27 16:04:11 29193:29193 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:11.3781768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:11.3782198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:11.3787698Z STAGE:2022-09-27 16:04:11 29191:29191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:11.3788177Z STAGE:2022-09-27 16:04:11 29192:29192 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:11.3811969Z STAGE:2022-09-27 16:04:11 29191:29191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:11.3813339Z STAGE:2022-09-27 16:04:11 29192:29192 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:11.3814074Z STAGE:2022-09-27 16:04:11 29193:29193 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:11.8031951Z ok (2.249s)
2022-09-27T16:04:11.8032201Z 
2022-09-27T16:04:11.8032705Z ----------------------------------------------------------------------
2022-09-27T16:04:11.8033179Z Ran 1 test in 2.250s
2022-09-27T16:04:11.8033310Z 
2022-09-27T16:04:11.8033372Z OK
2022-09-27T16:04:11.8033464Z 
2022-09-27T16:04:11.8033553Z Generating XML reports...
2022-09-27T16:04:11.8077827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160409.xml
2022-09-27T16:04:13.1089283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:13.1089703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:13.1097364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:13.1097887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:13.4489107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:13.4497856Z 
2022-09-27T16:04:13.4497995Z Running tests...
2022-09-27T16:04:13.4498741Z ----------------------------------------------------------------------
2022-09-27T16:04:13.6851448Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29247
2022-09-27T16:04:13.6877665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29248
2022-09-27T16:04:13.6904700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29249
2022-09-27T16:04:14.7726124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:14.7726778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:14.7789648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:14.7790052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:14.8070446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:14.8070870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:14.8133846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:14.8134488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:14.8192825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:14.8193313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:14.8508438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:14.8508813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:15.1685069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:15.1785953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:15.1786605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:15.1787682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:15.1788534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:15.1789131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:15.1893037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:15.1896617Z STAGE:2022-09-27 16:04:15 29249:29249 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:15.2796719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:15.2797663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:15.2800329Z STAGE:2022-09-27 16:04:15 29247:29247 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:15.2801362Z STAGE:2022-09-27 16:04:15 29248:29248 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:15.2826242Z STAGE:2022-09-27 16:04:15 29249:29249 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:15.2827431Z STAGE:2022-09-27 16:04:15 29247:29247 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:15.2828608Z STAGE:2022-09-27 16:04:15 29248:29248 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:15.5954021Z ok (2.145s)
2022-09-27T16:04:15.5954285Z 
2022-09-27T16:04:15.5954806Z ----------------------------------------------------------------------
2022-09-27T16:04:15.5955079Z Ran 1 test in 2.146s
2022-09-27T16:04:15.5955195Z 
2022-09-27T16:04:15.5955243Z OK
2022-09-27T16:04:15.5955335Z 
2022-09-27T16:04:15.5955635Z Generating XML reports...
2022-09-27T16:04:15.5999857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160413.xml
2022-09-27T16:04:16.9106902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:16.9107530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:16.9115851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:16.9116503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:17.2516710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:17.2526691Z 
2022-09-27T16:04:17.2527043Z Running tests...
2022-09-27T16:04:17.2527668Z ----------------------------------------------------------------------
2022-09-27T16:04:17.2543252Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-09-27T16:04:17.2543593Z 
2022-09-27T16:04:17.2544155Z ----------------------------------------------------------------------
2022-09-27T16:04:17.2544643Z Ran 1 test in 0.002s
2022-09-27T16:04:17.2544826Z 
2022-09-27T16:04:17.2544949Z OK (skipped=1)
2022-09-27T16:04:17.2545125Z 
2022-09-27T16:04:17.2545263Z Generating XML reports...
2022-09-27T16:04:17.2575850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml
2022-09-27T16:04:18.4693272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:18.4693913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:18.4702761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:18.4703386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:18.8086247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:18.8096502Z 
2022-09-27T16:04:18.8096857Z Running tests...
2022-09-27T16:04:18.8097475Z ----------------------------------------------------------------------
2022-09-27T16:04:18.8113728Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:04:18.8114201Z 
2022-09-27T16:04:18.8114535Z ----------------------------------------------------------------------
2022-09-27T16:04:18.8114832Z Ran 1 test in 0.002s
2022-09-27T16:04:18.8114988Z 
2022-09-27T16:04:18.8115099Z OK (skipped=1)
2022-09-27T16:04:18.8115216Z 
2022-09-27T16:04:18.8115303Z Generating XML reports...
2022-09-27T16:04:18.8144864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160418.xml
2022-09-27T16:04:20.0261748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:20.0262123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:20.0270089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:20.0270439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:20.3650067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:20.3659690Z 
2022-09-27T16:04:20.3660151Z Running tests...
2022-09-27T16:04:20.3660578Z ----------------------------------------------------------------------
2022-09-27T16:04:20.3677880Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:04:20.3678195Z 
2022-09-27T16:04:20.3678442Z ----------------------------------------------------------------------
2022-09-27T16:04:20.3679036Z Ran 1 test in 0.002s
2022-09-27T16:04:20.3679246Z 
2022-09-27T16:04:20.3679380Z OK (skipped=1)
2022-09-27T16:04:20.3679559Z 
2022-09-27T16:04:20.3679721Z Generating XML reports...
2022-09-27T16:04:20.3709856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160420.xml
2022-09-27T16:04:21.5841198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:21.5841627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:21.5851377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:21.5851843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:21.9225352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:21.9234684Z 
2022-09-27T16:04:21.9234928Z Running tests...
2022-09-27T16:04:21.9235562Z ----------------------------------------------------------------------
2022-09-27T16:04:22.1597564Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29333
2022-09-27T16:04:22.1625130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29334
2022-09-27T16:04:22.1652004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29335
2022-09-27T16:04:23.2479112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:23.2479781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:23.2713012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:23.2713637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:23.2811462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:23.2812171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:23.3040767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:23.3041419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:23.3292017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:23.3292747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:23.3610256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:23.3610890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:23.6801489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:23.6802241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:23.6802702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:23.6803339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:23.6803876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:23.6804404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:23.6810686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:23.6811661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:23.6812513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:23.6815867Z STAGE:2022-09-27 16:04:23 29333:29333 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:23.6816855Z STAGE:2022-09-27 16:04:23 29335:29335 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:23.6818808Z STAGE:2022-09-27 16:04:23 29334:29334 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:23.6841558Z STAGE:2022-09-27 16:04:23 29335:29335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:23.6843545Z STAGE:2022-09-27 16:04:23 29333:29333 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:23.6845404Z STAGE:2022-09-27 16:04:23 29334:29334 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:23.9699686Z ok (2.046s)
2022-09-27T16:04:23.9699976Z 
2022-09-27T16:04:23.9700410Z ----------------------------------------------------------------------
2022-09-27T16:04:23.9700680Z Ran 1 test in 2.046s
2022-09-27T16:04:23.9700785Z 
2022-09-27T16:04:23.9700847Z OK
2022-09-27T16:04:23.9700941Z 
2022-09-27T16:04:23.9701037Z Generating XML reports...
2022-09-27T16:04:23.9748195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160421.xml
2022-09-27T16:04:25.2650805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:25.2651191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:25.2658769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:25.2659145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:25.6038444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:25.6047545Z 
2022-09-27T16:04:25.6047787Z Running tests...
2022-09-27T16:04:25.6048436Z ----------------------------------------------------------------------
2022-09-27T16:04:25.8397976Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29389
2022-09-27T16:04:25.8424880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29390
2022-09-27T16:04:25.8451924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29391
2022-09-27T16:04:26.8991997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:26.8992684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:26.9280767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:26.9281448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:26.9305666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:26.9306357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:26.9613306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:26.9613935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:26.9657905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:26.9658531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:26.9981046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:26.9981690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:27.3155674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:27.3180501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:27.3181065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:27.3181685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:27.3182218Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:27.3256329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:27.3287175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:27.3287740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:27.4265357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:27.7504811Z ok (2.145s)
2022-09-27T16:04:27.7504999Z 
2022-09-27T16:04:27.7512297Z ----------------------------------------------------------------------
2022-09-27T16:04:27.7512654Z Ran 1 test in 2.146s
2022-09-27T16:04:27.7512778Z 
2022-09-27T16:04:27.7512842Z OK
2022-09-27T16:04:27.7512936Z 
2022-09-27T16:04:27.7513034Z Generating XML reports...
2022-09-27T16:04:27.7551285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160425.xml
2022-09-27T16:04:29.0519476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:29.0519855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:29.0527230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:29.0527604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:29.3884671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:29.3894165Z 
2022-09-27T16:04:29.3894292Z Running tests...
2022-09-27T16:04:29.3894714Z ----------------------------------------------------------------------
2022-09-27T16:04:29.6257174Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29442
2022-09-27T16:04:29.6283576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29443
2022-09-27T16:04:29.6310945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29444
2022-09-27T16:04:30.7034482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:30.7035190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:30.7035973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:30.7036574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:30.7390912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:30.7391386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:30.7394010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:30.7394647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:30.7439975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:30.7440632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:30.7767127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:30.7767771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.1176727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:31.1276951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:31.1277683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:31.1278497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:31.1279039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:31.1279583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:31.1383696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:31.1387240Z STAGE:2022-09-27 16:04:31 29443:29443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:31.2288797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:31.2289340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:31.2292666Z STAGE:2022-09-27 16:04:31 29442:29442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:31.2293463Z STAGE:2022-09-27 16:04:31 29444:29444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:31.2308451Z STAGE:2022-09-27 16:04:31 29442:29442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:31.2310692Z STAGE:2022-09-27 16:04:31 29443:29443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:31.2312298Z STAGE:2022-09-27 16:04:31 29444:29444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:31.5363124Z ok (2.147s)
2022-09-27T16:04:31.5363388Z 
2022-09-27T16:04:31.5363739Z ----------------------------------------------------------------------
2022-09-27T16:04:31.5364016Z Ran 1 test in 2.147s
2022-09-27T16:04:31.5364177Z 
2022-09-27T16:04:31.5364276Z OK
2022-09-27T16:04:31.5364362Z 
2022-09-27T16:04:31.5364459Z Generating XML reports...
2022-09-27T16:04:31.5409789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160429.xml
2022-09-27T16:04:32.8487140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:32.8487528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:32.8494850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:32.8495241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:33.1879728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:33.1888858Z 
2022-09-27T16:04:33.1889325Z Running tests...
2022-09-27T16:04:33.1889739Z ----------------------------------------------------------------------
2022-09-27T16:04:33.4255220Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29498
2022-09-27T16:04:33.4281990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29499
2022-09-27T16:04:33.4309518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29500
2022-09-27T16:04:34.4897193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:34.4898154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:34.5012330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:34.5012953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:34.5241867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:34.5242343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:34.5297990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:34.5298472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:34.5367504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:34.5368079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:34.5624008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:34.5624628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:34.8991049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:34.9091776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:34.9092533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:34.9093469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:34.9094034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:34.9094580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:34.9100343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:34.9101235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:34.9104065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:34.9106990Z STAGE:2022-09-27 16:04:34 29498:29498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:34.9107685Z STAGE:2022-09-27 16:04:34 29500:29500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:34.9108885Z STAGE:2022-09-27 16:04:34 29499:29499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:34.9121354Z STAGE:2022-09-27 16:04:34 29498:29498 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:34.9124720Z STAGE:2022-09-27 16:04:34 29500:29500 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:34.9125371Z STAGE:2022-09-27 16:04:34 29499:29499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:35.2357524Z ok (2.046s)
2022-09-27T16:04:35.2357746Z 
2022-09-27T16:04:35.2358262Z ----------------------------------------------------------------------
2022-09-27T16:04:35.2358696Z Ran 1 test in 2.047s
2022-09-27T16:04:35.2358874Z 
2022-09-27T16:04:35.2358940Z OK
2022-09-27T16:04:35.2359035Z 
2022-09-27T16:04:35.2359132Z Generating XML reports...
2022-09-27T16:04:35.2404576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160433.xml
2022-09-27T16:04:36.5414888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:36.5415262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:36.5423070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:36.5423423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:36.8864719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:36.8874163Z 
2022-09-27T16:04:36.8874279Z Running tests...
2022-09-27T16:04:36.8874732Z ----------------------------------------------------------------------
2022-09-27T16:04:37.1233383Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29554
2022-09-27T16:04:37.1260130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29555
2022-09-27T16:04:37.1287120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29556
2022-09-27T16:04:38.2484323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:38.2484993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:38.2487230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:38.2487832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:38.2765256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:38.2765935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:38.2839617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:38.2840212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:38.2848776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:38.2849316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:38.3102085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:38.3102710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:38.6336251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:38.6436943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:38.6437590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:38.6438501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:38.6439435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:38.6440066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:38.6543138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:38.7447070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:38.7447595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:39.1339416Z ok (2.246s)
2022-09-27T16:04:39.1339656Z 
2022-09-27T16:04:39.1340162Z ----------------------------------------------------------------------
2022-09-27T16:04:39.1340595Z Ran 1 test in 2.246s
2022-09-27T16:04:39.1340767Z 
2022-09-27T16:04:39.1340832Z OK
2022-09-27T16:04:39.1340932Z 
2022-09-27T16:04:39.1341029Z Generating XML reports...
2022-09-27T16:04:39.1385981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160436.xml
2022-09-27T16:04:40.4362793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:40.4363185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:40.4369523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:40.4369872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:40.7736372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:40.7745273Z 
2022-09-27T16:04:40.7745394Z Running tests...
2022-09-27T16:04:40.7746061Z ----------------------------------------------------------------------
2022-09-27T16:04:41.0151300Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29631
2022-09-27T16:04:41.0178443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29632
2022-09-27T16:04:41.0205762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29633
2022-09-27T16:04:42.1340912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:42.1341632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:42.1412748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:42.1413413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:42.1438883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:42.1439528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:42.1690761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:42.1691409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:42.1772535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:42.1773175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:42.1795068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:42.1795709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:42.5145492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:42.5246642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:42.5247302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:42.5248078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:42.5248621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:42.5249491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:42.5254908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:42.5255484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:42.5256360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:42.8255349Z skip: CUDA is not available. (2.051s)
2022-09-27T16:04:42.8255686Z 
2022-09-27T16:04:42.8256165Z ----------------------------------------------------------------------
2022-09-27T16:04:42.8256408Z Ran 1 test in 2.051s
2022-09-27T16:04:42.8256526Z 
2022-09-27T16:04:42.8256810Z OK (skipped=1)
2022-09-27T16:04:42.8256923Z 
2022-09-27T16:04:42.8257010Z Generating XML reports...
2022-09-27T16:04:42.8302040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160440.xml
2022-09-27T16:04:44.1244818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:44.1245478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:44.1253716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:44.1254346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:44.4628101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:44.4638371Z 
2022-09-27T16:04:44.4638656Z Running tests...
2022-09-27T16:04:44.4639309Z ----------------------------------------------------------------------
2022-09-27T16:04:44.6994222Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29684
2022-09-27T16:04:44.7020605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29685
2022-09-27T16:04:44.7047585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29686
2022-09-27T16:04:45.8044998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:45.8045402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:45.8295269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:45.8295833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:45.8384297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:45.8384883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:45.8402672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:45.8403213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:45.8644020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:45.8644561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:45.8740827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:45.8741336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:46.2053880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:46.2054653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:46.2055055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:46.2055683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:46.2056223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:46.2056738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:46.2160362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:46.2161114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:46.3067188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:46.6096050Z skip: Need at least 2 CUDA devices (2.145s)
2022-09-27T16:04:46.6096261Z 
2022-09-27T16:04:46.6096753Z ----------------------------------------------------------------------
2022-09-27T16:04:46.6097463Z Ran 1 test in 2.146s
2022-09-27T16:04:46.6097586Z 
2022-09-27T16:04:46.6097669Z OK (skipped=1)
2022-09-27T16:04:46.6097780Z 
2022-09-27T16:04:46.6097855Z Generating XML reports...
2022-09-27T16:04:46.6143009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160444.xml
2022-09-27T16:04:47.9127204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:47.9127592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:47.9135381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:47.9135758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:48.2520227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:48.2529436Z 
2022-09-27T16:04:48.2529564Z Running tests...
2022-09-27T16:04:48.2530136Z ----------------------------------------------------------------------
2022-09-27T16:04:48.4890985Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29737
2022-09-27T16:04:48.4916995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29738
2022-09-27T16:04:48.4944031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29739
2022-09-27T16:04:49.5281558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:49.5282234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:49.5596517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:49.5693218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:49.5694146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:49.5694720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:49.5696956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:49.5697552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:49.6042631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:49.6043292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:49.6046799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:49.6047197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:49.9212243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:49.9312227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:49.9312940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:49.9313619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:49.9314142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:49.9314668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:49.9420021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:49.9420694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:49.9480079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6ok9iuk
2022-09-27T16:04:49.9480755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkac88ms
2022-09-27T16:04:49.9482065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6ok9iuk/_remote_module_non_scriptable.py
2022-09-27T16:04:49.9482684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkac88ms/_remote_module_non_scriptable.py
2022-09-27T16:04:50.0322728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:50.0385849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7znvxjwd
2022-09-27T16:04:50.0387814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7znvxjwd/_remote_module_non_scriptable.py
2022-09-27T16:04:50.3996551Z ok (2.146s)
2022-09-27T16:04:50.3996779Z 
2022-09-27T16:04:50.3997229Z ----------------------------------------------------------------------
2022-09-27T16:04:50.3997725Z Ran 1 test in 2.147s
2022-09-27T16:04:50.3997951Z 
2022-09-27T16:04:50.3998050Z OK
2022-09-27T16:04:50.3998164Z 
2022-09-27T16:04:50.3999221Z Generating XML reports...
2022-09-27T16:04:50.4043565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160448.xml
2022-09-27T16:04:51.7178619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:51.7178992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:51.7186914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:51.7187297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:52.0578484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:52.0587704Z 
2022-09-27T16:04:52.0587845Z Running tests...
2022-09-27T16:04:52.0588276Z ----------------------------------------------------------------------
2022-09-27T16:04:52.2940533Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29790
2022-09-27T16:04:52.2967418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29791
2022-09-27T16:04:52.2995085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29792
2022-09-27T16:04:53.3604126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:53.3604808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:53.3851121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:53.3851733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:53.3952947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:53.3953578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:53.4201307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:53.4201949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:53.4226376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:53.4226987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:53.4540335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:53.4541033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:53.7762926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:53.7797180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:53.7797729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:53.7798384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:53.7798967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:53.7864844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:53.7905158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:53.7905843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:53.8879323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:54.1044298Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T16:04:54.1044530Z 
2022-09-27T16:04:54.1045014Z ----------------------------------------------------------------------
2022-09-27T16:04:54.1045468Z Ran 1 test in 2.046s
2022-09-27T16:04:54.1045629Z 
2022-09-27T16:04:54.1045706Z OK (skipped=1)
2022-09-27T16:04:54.1045803Z 
2022-09-27T16:04:54.1045889Z Generating XML reports...
2022-09-27T16:04:54.1090478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160452.xml
2022-09-27T16:04:55.3868134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:55.3868533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:55.3876579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:55.3876951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:55.7244680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:55.7254481Z 
2022-09-27T16:04:55.7254627Z Running tests...
2022-09-27T16:04:55.7254961Z ----------------------------------------------------------------------
2022-09-27T16:04:55.9624287Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29843
2022-09-27T16:04:55.9651136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29844
2022-09-27T16:04:55.9677665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29845
2022-09-27T16:04:57.0404139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.0404790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.0728438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.0729062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.0737652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.0738237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.0797936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.0798577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.1088191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.1088857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.1146074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.1146668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.4658985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:57.4760258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:57.4760900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:04:57.4761721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:57.4762337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:57.4763288Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:04:57.4769265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:57.4769762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:57.4770150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:04:57.7724512Z skip: Need at least 2 CUDA devices (2.047s)
2022-09-27T16:04:57.7724845Z 
2022-09-27T16:04:57.7725222Z ----------------------------------------------------------------------
2022-09-27T16:04:57.7725480Z Ran 1 test in 2.047s
2022-09-27T16:04:57.7725598Z 
2022-09-27T16:04:57.7725661Z OK (skipped=1)
2022-09-27T16:04:57.7725786Z 
2022-09-27T16:04:57.7725873Z Generating XML reports...
2022-09-27T16:04:57.7772677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160455.xml
2022-09-27T16:04:59.0786291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:59.0786687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:59.0794668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:59.0795173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:59.4167798Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:59.4176593Z 
2022-09-27T16:04:59.4176742Z Running tests...
2022-09-27T16:04:59.4177311Z ----------------------------------------------------------------------
2022-09-27T16:04:59.4194090Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-09-27T16:04:59.4194625Z 
2022-09-27T16:04:59.4194850Z ----------------------------------------------------------------------
2022-09-27T16:04:59.4195103Z Ran 1 test in 0.002s
2022-09-27T16:04:59.4195220Z 
2022-09-27T16:04:59.4195281Z OK (skipped=1)
2022-09-27T16:04:59.4195394Z 
2022-09-27T16:04:59.4195481Z Generating XML reports...
2022-09-27T16:04:59.4224035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160459.xml
2022-09-27T16:05:00.6090639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:00.6091016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:00.6099217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:00.6099608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:00.9469159Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:00.9478383Z 
2022-09-27T16:05:00.9478522Z Running tests...
2022-09-27T16:05:00.9479410Z ----------------------------------------------------------------------
2022-09-27T16:05:00.9494754Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-09-27T16:05:00.9495203Z 
2022-09-27T16:05:00.9495777Z ----------------------------------------------------------------------
2022-09-27T16:05:00.9496250Z Ran 1 test in 0.002s
2022-09-27T16:05:00.9496448Z 
2022-09-27T16:05:00.9496598Z OK (skipped=1)
2022-09-27T16:05:00.9496741Z 
2022-09-27T16:05:00.9496831Z Generating XML reports...
2022-09-27T16:05:00.9525995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160500.xml
2022-09-27T16:05:01.3790390Z Running distributed/rpc/test_faulty_agent ... [2022-09-27 16:05:01.378563]
2022-09-27T16:05:01.3791074Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:05:01.378621]
2022-09-27T16:05:02.5389330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4waeypbh
2022-09-27T16:05:02.5389928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4waeypbh/_remote_module_non_scriptable.py
2022-09-27T16:05:02.7918767Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]>
2022-09-27T16:05:02.7919608Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest)
2022-09-27T16:05:02.7920067Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest)
2022-09-27T16:05:02.7922923Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]>
2022-09-27T16:05:02.7925451Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7926083Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7926417Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7926717Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7927112Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7927398Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7927708Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7928046Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7928380Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7928695Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7929032Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7929373Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7929675Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7929968Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7930265Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7930556Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7930886Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7931218Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7931555Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7931863Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest)
2022-09-27T16:05:02.7932700Z <unittest.suite.TestSuite tests=[<__main__.FaultyJitFaultyAgentRpcTest testMethod=test_remote_timeout_to_here_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]>
2022-09-27T16:05:02.7933511Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:02.7933840Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:02.7934151Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:02.7934476Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:02.7934789Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:02.7935108Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest)
2022-09-27T16:05:03.8006222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:03.8006893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:03.8015229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:03.8015865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:03.9529022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6ccbmjk
2022-09-27T16:05:03.9530044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6ccbmjk/_remote_module_non_scriptable.py
2022-09-27T16:05:04.2045521Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:05:04.2056674Z 
2022-09-27T16:05:04.2057080Z Running tests...
2022-09-27T16:05:04.2057470Z ----------------------------------------------------------------------
2022-09-27T16:05:04.4381791Z   test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29926
2022-09-27T16:05:04.4408535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29927
2022-09-27T16:05:04.4436327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29928
2022-09-27T16:05:04.4462728Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29929
2022-09-27T16:05:05.5955844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:05.5956458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:05.6146989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:05.6147589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:05.6160279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:05.6160842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:05.6303015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:05.6303589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.6384951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:05.6514934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:05.6515868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:05.6516502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.6528953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:05.6529553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.6724862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:05.6725457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.7563406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9o9qbji
2022-09-27T16:05:05.7564151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9o9qbji/_remote_module_non_scriptable.py
2022-09-27T16:05:05.7829120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvli9_j9c
2022-09-27T16:05:05.7830639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvli9_j9c/_remote_module_non_scriptable.py
2022-09-27T16:05:05.7969881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cd3ghwb
2022-09-27T16:05:05.7971026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cd3ghwb/_remote_module_non_scriptable.py
2022-09-27T16:05:05.7983027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpis0izdyk
2022-09-27T16:05:05.7985070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpis0izdyk/_remote_module_non_scriptable.py
2022-09-27T16:05:06.0079017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:06.0480470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:05:06.0495152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:05:06.0520557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:06.1283690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:06.1386158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:05:06.1387439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:06.1388530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:05:06.1389848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:05:06.1390419Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:05:06.1391173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:05:06.1396313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:05:13.6624106Z ok (9.456s)
2022-09-27T16:05:13.6624614Z 
2022-09-27T16:05:13.6625140Z ----------------------------------------------------------------------
2022-09-27T16:05:13.6625405Z Ran 1 test in 9.457s
2022-09-27T16:05:13.6625524Z 
2022-09-27T16:05:13.6625580Z OK
2022-09-27T16:05:13.6625673Z 
2022-09-27T16:05:13.6625769Z Generating XML reports...
2022-09-27T16:05:13.6670616Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220927160504.xml
2022-09-27T16:05:14.9614514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:14.9614884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:14.9622534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:14.9622985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:15.1148941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rfkjdc5
2022-09-27T16:05:15.1150341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rfkjdc5/_remote_module_non_scriptable.py
2022-09-27T16:05:15.3657867Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:05:15.3669038Z 
2022-09-27T16:05:15.3669187Z Running tests...
2022-09-27T16:05:15.3669691Z ----------------------------------------------------------------------
2022-09-27T16:05:15.6003313Z   test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30113
2022-09-27T16:05:15.6030144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30114
2022-09-27T16:05:15.6057560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30115
2022-09-27T16:05:15.6085258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30116
2022-09-27T16:05:16.7262638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:16.7263310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:16.7588414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:16.7589093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:16.8019556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:16.8020218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:16.8235921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:16.8236596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:16.8356840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:16.8357367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:16.8361708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:16.8362394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:16.8592397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:16.8592958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:16.8701813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:16.8702486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:16.8852046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8o5eclg
2022-09-27T16:05:16.8853267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8o5eclg/_remote_module_non_scriptable.py
2022-09-27T16:05:16.9622485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsryjsrd
2022-09-27T16:05:16.9624033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsryjsrd/_remote_module_non_scriptable.py
2022-09-27T16:05:16.9839498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt00jvmg3
2022-09-27T16:05:16.9840421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt00jvmg3/_remote_module_non_scriptable.py
2022-09-27T16:05:16.9993231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtjo61po
2022-09-27T16:05:16.9994410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtjo61po/_remote_module_non_scriptable.py
2022-09-27T16:05:17.1366334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:17.2147842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:05:17.2345158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:05:17.2665754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:23.7228154Z ok (8.356s)
2022-09-27T16:05:23.7228333Z 
2022-09-27T16:05:23.7228746Z ----------------------------------------------------------------------
2022-09-27T16:05:23.7229031Z Ran 1 test in 8.356s
2022-09-27T16:05:23.7229154Z 
2022-09-27T16:05:23.7229216Z OK
2022-09-27T16:05:23.7229309Z 
2022-09-27T16:05:23.7229404Z Generating XML reports...
2022-09-27T16:05:23.7277177Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220927160515.xml
2022-09-27T16:05:25.0376217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:25.0376611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:25.0384114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:25.0384571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:25.1905734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcvnzpoo
2022-09-27T16:05:25.1906298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcvnzpoo/_remote_module_non_scriptable.py
2022-09-27T16:05:25.4415959Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:05:25.4426509Z 
2022-09-27T16:05:25.4426643Z Running tests...
2022-09-27T16:05:25.4427544Z ----------------------------------------------------------------------
2022-09-27T16:05:25.6771149Z   test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30288
2022-09-27T16:05:25.6797715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30289
2022-09-27T16:05:25.6824585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30290
2022-09-27T16:05:25.6852950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30291
2022-09-27T16:05:26.8135599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:26.8136242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:26.8354812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:26.8355411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:26.8471591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:26.8472293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:26.8686241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:26.8686926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:26.8909171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:26.8909785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:26.9112912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:26.9113582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:26.9244851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:26.9245428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:26.9469004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:26.9469647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:26.9924009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4i3u4bex
2022-09-27T16:05:26.9924773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4i3u4bex/_remote_module_non_scriptable.py
2022-09-27T16:05:26.9971982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lu6i6n4
2022-09-27T16:05:26.9973271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lu6i6n4/_remote_module_non_scriptable.py
2022-09-27T16:05:27.0497798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4uo81yk
2022-09-27T16:05:27.0499075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4uo81yk/_remote_module_non_scriptable.py
2022-09-27T16:05:27.0773920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhhinsoe
2022-09-27T16:05:27.0775178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhhinsoe/_remote_module_non_scriptable.py
2022-09-27T16:05:27.2441741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:27.2481698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:05:27.3010721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:27.3477048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:05:33.6994293Z ok (8.256s)
2022-09-27T16:05:33.6994554Z 
2022-09-27T16:05:33.6994916Z ----------------------------------------------------------------------
2022-09-27T16:05:33.6995155Z Ran 1 test in 8.257s
2022-09-27T16:05:33.6995283Z 
2022-09-27T16:05:33.6995343Z OK
2022-09-27T16:05:33.6995663Z 
2022-09-27T16:05:33.6995758Z Generating XML reports...
2022-09-27T16:05:33.7040486Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160525.xml
2022-09-27T16:05:35.0211257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:35.0211640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:35.0218930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:35.0219281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:35.1733867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5_ght1r
2022-09-27T16:05:35.1734919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5_ght1r/_remote_module_non_scriptable.py
2022-09-27T16:05:35.4243182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:05:35.4253540Z 
2022-09-27T16:05:35.4253646Z Running tests...
2022-09-27T16:05:35.4254263Z ----------------------------------------------------------------------
2022-09-27T16:05:35.6600766Z   test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30463
2022-09-27T16:05:35.6626931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30464
2022-09-27T16:05:35.6653355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30465
2022-09-27T16:05:35.6681149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30466
2022-09-27T16:05:36.7536988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:36.7537620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:36.7805402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:36.7806052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:36.7868901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:36.7869527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:36.8139984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:36.8140663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:36.8756458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:36.8757136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:36.9085937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:36.9086547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:36.9135840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdudut_rd
2022-09-27T16:05:36.9137232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdudut_rd/_remote_module_non_scriptable.py
2022-09-27T16:05:36.9229115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:36.9229707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:36.9395757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw580hf0n
2022-09-27T16:05:36.9397083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw580hf0n/_remote_module_non_scriptable.py
2022-09-27T16:05:36.9565442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:37.0344329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:37.0345343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9ycrolq
2022-09-27T16:05:37.0346032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9ycrolq/_remote_module_non_scriptable.py
2022-09-27T16:05:37.0836429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppteyf0bi
2022-09-27T16:05:37.0837226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppteyf0bi/_remote_module_non_scriptable.py
2022-09-27T16:05:37.1676099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:05:37.1921374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:37.2851011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:37.3546882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:05:43.7828147Z ok (8.357s)
2022-09-27T16:05:43.7828334Z 
2022-09-27T16:05:43.7828697Z ----------------------------------------------------------------------
2022-09-27T16:05:43.7828954Z Ran 1 test in 8.357s
2022-09-27T16:05:43.7830743Z 
2022-09-27T16:05:43.7830960Z OK
2022-09-27T16:05:43.7831184Z 
2022-09-27T16:05:43.7831373Z Generating XML reports...
2022-09-27T16:05:43.7879684Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160535.xml
2022-09-27T16:05:45.0930517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:45.0930895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:45.0938741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:45.0939088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:45.2452846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyojbsomo
2022-09-27T16:05:45.2453732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyojbsomo/_remote_module_non_scriptable.py
2022-09-27T16:05:45.4951945Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:05:45.4962407Z 
2022-09-27T16:05:45.4962545Z Running tests...
2022-09-27T16:05:45.4962949Z ----------------------------------------------------------------------
2022-09-27T16:05:45.7292281Z   test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30638
2022-09-27T16:05:45.7319432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30639
2022-09-27T16:05:45.7346087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30640
2022-09-27T16:05:45.7374248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30641
2022-09-27T16:05:46.8493796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.8494454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.8817943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.8818578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:46.8912177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.8912791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.9240387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.9241034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:46.9254878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.9255710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.9451595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.9452200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.9606828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.9607508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:46.9799419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.9800100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:47.0079070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzyuwgve
2022-09-27T16:05:47.0079906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzyuwgve/_remote_module_non_scriptable.py
2022-09-27T16:05:47.0502723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc966g2dn
2022-09-27T16:05:47.0503445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc966g2dn/_remote_module_non_scriptable.py
2022-09-27T16:05:47.0859460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf3pguyx
2022-09-27T16:05:47.0860607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf3pguyx/_remote_module_non_scriptable.py
2022-09-27T16:05:47.1099069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2w1td94
2022-09-27T16:05:47.1099789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2w1td94/_remote_module_non_scriptable.py
2022-09-27T16:05:47.2798349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:05:47.3029451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:47.3361488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:05:47.3783690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:58.3583940Z ok (12.862s)
2022-09-27T16:05:58.3584252Z 
2022-09-27T16:05:58.3584823Z ----------------------------------------------------------------------
2022-09-27T16:05:58.3585135Z Ran 1 test in 12.862s
2022-09-27T16:05:58.3585254Z 
2022-09-27T16:05:58.3585317Z OK
2022-09-27T16:05:58.3585395Z 
2022-09-27T16:05:58.3585491Z Generating XML reports...
2022-09-27T16:05:58.3632606Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160545.xml
2022-09-27T16:05:59.6850704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:59.6851098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:59.6858911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:59.6859362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:59.8373958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeoofeqa8
2022-09-27T16:05:59.8374752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeoofeqa8/_remote_module_non_scriptable.py
2022-09-27T16:06:00.0885105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:00.0895301Z 
2022-09-27T16:06:00.0895412Z Running tests...
2022-09-27T16:06:00.0896090Z ----------------------------------------------------------------------
2022-09-27T16:06:00.3206945Z   test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30813
2022-09-27T16:06:00.3234069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30814
2022-09-27T16:06:00.3261152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30815
2022-09-27T16:06:00.3290204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30816
2022-09-27T16:06:01.4495522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:01.4496156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:01.4664721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:01.4665351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:01.4741155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:01.4741782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:01.4749852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:01.4750502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:01.4841314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:01.4841939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:01.5035983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:01.5036656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:01.5108367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:01.5121429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:01.5122274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:01.5122852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:01.6092540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w49b0qs
2022-09-27T16:06:01.6093381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w49b0qs/_remote_module_non_scriptable.py
2022-09-27T16:06:01.6286770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu_y687_
2022-09-27T16:06:01.6288146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu_y687_/_remote_module_non_scriptable.py
2022-09-27T16:06:01.6347634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlgssiib
2022-09-27T16:06:01.6349715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlgssiib/_remote_module_non_scriptable.py
2022-09-27T16:06:01.6415833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc6ypl_u
2022-09-27T16:06:01.6417714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc6ypl_u/_remote_module_non_scriptable.py
2022-09-27T16:06:01.8617575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:01.8820763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:01.8853979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:01.9128943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:08.3435242Z ok (8.254s)
2022-09-27T16:06:08.3435508Z 
2022-09-27T16:06:08.3436265Z ----------------------------------------------------------------------
2022-09-27T16:06:08.3436530Z Ran 1 test in 8.254s
2022-09-27T16:06:08.3436649Z 
2022-09-27T16:06:08.3437017Z OK
2022-09-27T16:06:08.3437207Z 
2022-09-27T16:06:08.3437301Z Generating XML reports...
2022-09-27T16:06:08.3485197Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160600.xml
2022-09-27T16:06:09.6676252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:09.6676756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:09.6683501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:09.6683865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:09.8201154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeysl8rnd
2022-09-27T16:06:09.8202077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeysl8rnd/_remote_module_non_scriptable.py
2022-09-27T16:06:10.0713071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:10.0723603Z 
2022-09-27T16:06:10.0723690Z Running tests...
2022-09-27T16:06:10.0724619Z ----------------------------------------------------------------------
2022-09-27T16:06:10.3040036Z   test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30988
2022-09-27T16:06:10.3066229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30989
2022-09-27T16:06:10.3092938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30990
2022-09-27T16:06:10.3121477Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30991
2022-09-27T16:06:11.4840823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:11.4841539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:11.5186265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:11.5186898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:11.5240349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:11.5241002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:11.5566230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:11.5566931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:11.5692923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:11.5693519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:11.6024753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:11.6025448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:11.6438724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08k3u4y8
2022-09-27T16:06:11.6439573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08k3u4y8/_remote_module_non_scriptable.py
2022-09-27T16:06:11.6603558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:11.6604204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:11.6813951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_ry4q18
2022-09-27T16:06:11.6815753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_ry4q18/_remote_module_non_scriptable.py
2022-09-27T16:06:11.6933442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:11.6934378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:11.7276506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhmizqry
2022-09-27T16:06:11.7278012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhmizqry/_remote_module_non_scriptable.py
2022-09-27T16:06:11.8209594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckee9j0v
2022-09-27T16:06:11.8210656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckee9j0v/_remote_module_non_scriptable.py
2022-09-27T16:06:11.8955757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:11.9323365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:11.9776053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:12.0868420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:12.6177563Z ok (2.545s)
2022-09-27T16:06:12.6177836Z 
2022-09-27T16:06:12.6178348Z ----------------------------------------------------------------------
2022-09-27T16:06:12.6178728Z Ran 1 test in 2.545s
2022-09-27T16:06:12.6178834Z 
2022-09-27T16:06:12.6178896Z OK
2022-09-27T16:06:12.6178989Z 
2022-09-27T16:06:12.6179084Z Generating XML reports...
2022-09-27T16:06:12.6224194Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160610.xml
2022-09-27T16:06:13.9575268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:13.9575939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:13.9584006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:13.9584533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:14.1104648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgnp7avg
2022-09-27T16:06:14.1105655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgnp7avg/_remote_module_non_scriptable.py
2022-09-27T16:06:14.3617526Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:14.3628198Z 
2022-09-27T16:06:14.3628296Z Running tests...
2022-09-27T16:06:14.3628956Z ----------------------------------------------------------------------
2022-09-27T16:06:14.5974865Z   test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31163
2022-09-27T16:06:14.6001096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31164
2022-09-27T16:06:14.6029178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31165
2022-09-27T16:06:14.6057522Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31166
2022-09-27T16:06:15.6845643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.6846257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.6860879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.6861529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.6925703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.6926342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.7073072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.7073632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.7214611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.7215267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.7233297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.7233931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.7296369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.7297037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.7420037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.8467001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.8467651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooqs73lv
2022-09-27T16:06:15.8468348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooqs73lv/_remote_module_non_scriptable.py
2022-09-27T16:06:15.8521531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyudczrj5
2022-09-27T16:06:15.8522903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyudczrj5/_remote_module_non_scriptable.py
2022-09-27T16:06:15.8545377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuu5x2ik
2022-09-27T16:06:15.8546756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuu5x2ik/_remote_module_non_scriptable.py
2022-09-27T16:06:15.8667436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo06l6hyr
2022-09-27T16:06:15.8669880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo06l6hyr/_remote_module_non_scriptable.py
2022-09-27T16:06:16.0976535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:16.1054968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:16.1200271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:16.1209105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:22.5198756Z ok (8.157s)
2022-09-27T16:06:22.5199012Z 
2022-09-27T16:06:22.5199478Z ----------------------------------------------------------------------
2022-09-27T16:06:22.5199870Z Ran 1 test in 8.157s
2022-09-27T16:06:22.5200061Z 
2022-09-27T16:06:22.5200158Z OK
2022-09-27T16:06:22.5200292Z 
2022-09-27T16:06:22.5200464Z Generating XML reports...
2022-09-27T16:06:22.5250510Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160614.xml
2022-09-27T16:06:23.8484046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:23.8484411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:23.8491783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:23.8492173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:24.0012199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgudhj89
2022-09-27T16:06:24.0013727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgudhj89/_remote_module_non_scriptable.py
2022-09-27T16:06:24.2518866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:24.2529346Z 
2022-09-27T16:06:24.2529470Z Running tests...
2022-09-27T16:06:24.2529899Z ----------------------------------------------------------------------
2022-09-27T16:06:24.4856384Z   test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31338
2022-09-27T16:06:24.4882149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31339
2022-09-27T16:06:24.4908421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31340
2022-09-27T16:06:24.4937248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31341
2022-09-27T16:06:25.5831494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.5832181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.5844264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.5844948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.6188065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.6188804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.6201914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.6202559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.6743363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.6744155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.6885761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.6886437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.7093333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.7094040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.7227184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.7227895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.7448630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxl_ph0m
2022-09-27T16:06:25.7449355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxl_ph0m/_remote_module_non_scriptable.py
2022-09-27T16:06:25.7455531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaigdtie5
2022-09-27T16:06:25.7456990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaigdtie5/_remote_module_non_scriptable.py
2022-09-27T16:06:25.8372036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7puc_67w
2022-09-27T16:06:25.8372993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7puc_67w/_remote_module_non_scriptable.py
2022-09-27T16:06:25.8480315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vxx9dor
2022-09-27T16:06:25.8481108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vxx9dor/_remote_module_non_scriptable.py
2022-09-27T16:06:25.9955458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:25.9974891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:26.0993920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:26.1060004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:31.5064090Z ok (7.253s)
2022-09-27T16:06:31.5064348Z 
2022-09-27T16:06:31.5064906Z ----------------------------------------------------------------------
2022-09-27T16:06:31.5065375Z Ran 1 test in 7.253s
2022-09-27T16:06:31.5065478Z 
2022-09-27T16:06:31.5065538Z OK
2022-09-27T16:06:31.5065628Z 
2022-09-27T16:06:31.5065722Z Generating XML reports...
2022-09-27T16:06:31.5111750Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160624.xml
2022-09-27T16:06:32.8356434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:32.8356813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:32.8364617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:32.8365065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:32.9891214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1o6ydq1
2022-09-27T16:06:32.9892196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1o6ydq1/_remote_module_non_scriptable.py
2022-09-27T16:06:33.2402972Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:33.2413765Z 
2022-09-27T16:06:33.2414165Z Running tests...
2022-09-27T16:06:33.2414592Z ----------------------------------------------------------------------
2022-09-27T16:06:33.4756520Z   test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31513
2022-09-27T16:06:33.4784287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31514
2022-09-27T16:06:33.4810605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31515
2022-09-27T16:06:33.4838529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31516
2022-09-27T16:06:34.5995379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:34.5996114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:34.6185735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:34.6186261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:34.6204517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:34.6205196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:34.6341560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:34.6342114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.6548602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:34.6549364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.6566611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:34.6567236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.6940227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:34.6940888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:34.7270745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:34.7271723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.7601078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgv09m8h
2022-09-27T16:06:34.7602612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgv09m8h/_remote_module_non_scriptable.py
2022-09-27T16:06:34.7823480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tdqv3gb
2022-09-27T16:06:34.7825490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tdqv3gb/_remote_module_non_scriptable.py
2022-09-27T16:06:34.7836998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7qkz0xq
2022-09-27T16:06:34.7838519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7qkz0xq/_remote_module_non_scriptable.py
2022-09-27T16:06:34.8513504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprabfyvyy
2022-09-27T16:06:34.8514871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprabfyvyy/_remote_module_non_scriptable.py
2022-09-27T16:06:35.0099455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:35.0341417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:35.0538870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:35.1029606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:37.1451011Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-09-27T16:06:37.1452441Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2022-09-27T16:06:37.1454180Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f5aa701b34b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:06:37.1455945Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f5aa7016d1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:06:37.1457807Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f5aa7018abe in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:06:37.1459487Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7f5aab7f9330 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1461756Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5cc (0x7f5aab7e932c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1464160Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7f5ab3822baa in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:06:37.1466690Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7f5aab7edc1c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1468917Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f5ab381dc45 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:06:37.1470557Z frame #8: <unknown function> + 0x4789fa3 (0x7f5aab7e6fa3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1472200Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f5aab7e7b38 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1474036Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f5aab7e2047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1475376Z frame #11: <unknown function> + 0x47b9a52 (0x7f5aab816a52 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:06:37.1476538Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f5aa700977b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:06:37.1477342Z frame #13: <unknown function> + 0xc9039 (0x7f5aa6f20039 in /opt/conda/lib/libstdc++.so.6)
2022-09-27T16:06:37.1478223Z frame #14: <unknown function> + 0x8609 (0x7f5abb6ad609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-09-27T16:06:37.1479055Z frame #15: clone + 0x43 (0x7f5abb5d2133 in /lib/x86_64-linux-gnu/libc.so.6)
2022-09-27T16:06:37.1479410Z 
2022-09-27T16:06:37.4920169Z ok (4.250s)
2022-09-27T16:06:37.4920328Z 
2022-09-27T16:06:37.4920713Z ----------------------------------------------------------------------
2022-09-27T16:06:37.4920966Z Ran 1 test in 4.250s
2022-09-27T16:06:37.4921099Z 
2022-09-27T16:06:37.4921191Z OK
2022-09-27T16:06:37.4921302Z 
2022-09-27T16:06:37.4921395Z Generating XML reports...
2022-09-27T16:06:37.4966057Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160633.xml
2022-09-27T16:06:38.8016734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:38.8017106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:38.8024617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:38.8024985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:38.9540345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_aimesqe
2022-09-27T16:06:38.9541324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_aimesqe/_remote_module_non_scriptable.py
2022-09-27T16:06:39.2039783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:39.2049806Z 
2022-09-27T16:06:39.2049938Z Running tests...
2022-09-27T16:06:39.2050979Z ----------------------------------------------------------------------
2022-09-27T16:06:39.4390287Z   test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31688
2022-09-27T16:06:39.4417506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31689
2022-09-27T16:06:39.4444731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31690
2022-09-27T16:06:39.4473053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31691
2022-09-27T16:06:40.5608297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.5609331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.5712054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.5712867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.5796131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.5796700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.5952104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.5952809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.5966767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.5967354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.6084899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.6085582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.6169186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.6169758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.6306819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.6307468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.7212494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5l5k7ca
2022-09-27T16:06:40.7213319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5l5k7ca/_remote_module_non_scriptable.py
2022-09-27T16:06:40.7337368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1tenzsj
2022-09-27T16:06:40.7338463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1tenzsj/_remote_module_non_scriptable.py
2022-09-27T16:06:40.7444870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm02rn9os
2022-09-27T16:06:40.7446354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm02rn9os/_remote_module_non_scriptable.py
2022-09-27T16:06:40.7553777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4pga1q_
2022-09-27T16:06:40.7555184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4pga1q_/_remote_module_non_scriptable.py
2022-09-27T16:06:40.9737603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:40.9887732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:41.0053689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:41.0340253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:47.4640812Z ok (8.259s)
2022-09-27T16:06:47.4641042Z 
2022-09-27T16:06:47.4641405Z ----------------------------------------------------------------------
2022-09-27T16:06:47.4641664Z Ran 1 test in 8.259s
2022-09-27T16:06:47.4641782Z 
2022-09-27T16:06:47.4641831Z OK
2022-09-27T16:06:47.4641923Z 
2022-09-27T16:06:47.4642016Z Generating XML reports...
2022-09-27T16:06:47.4688309Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160639.xml
2022-09-27T16:06:48.7682893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:48.7683272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:48.7691514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:48.7691927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:48.9215370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09apqjzz
2022-09-27T16:06:48.9216371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09apqjzz/_remote_module_non_scriptable.py
2022-09-27T16:06:49.1728219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:49.1739080Z 
2022-09-27T16:06:49.1739250Z Running tests...
2022-09-27T16:06:49.1739739Z ----------------------------------------------------------------------
2022-09-27T16:06:49.4120184Z   test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31863
2022-09-27T16:06:49.4146278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31864
2022-09-27T16:06:49.4172990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31865
2022-09-27T16:06:49.4201422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31866
2022-09-27T16:06:50.5532429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.5533090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.5854204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.5854829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.5921713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.5922340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.6006520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.6007136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.6081803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.6082420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.6289717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.6290349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.6375616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.6376257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.6445649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.6446271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.7115826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm89ga26k
2022-09-27T16:06:50.7116985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm89ga26k/_remote_module_non_scriptable.py
2022-09-27T16:06:50.7585767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp529msn_q
2022-09-27T16:06:50.7588040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp529msn_q/_remote_module_non_scriptable.py
2022-09-27T16:06:50.7630360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajgg243t
2022-09-27T16:06:50.7632259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajgg243t/_remote_module_non_scriptable.py
2022-09-27T16:06:50.7692052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf0k3lkt
2022-09-27T16:06:50.7694294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf0k3lkt/_remote_module_non_scriptable.py
2022-09-27T16:06:50.9644833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:51.0163369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:06:51.0186022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:51.0269826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:06:57.4345940Z ok (8.260s)
2022-09-27T16:06:57.4346168Z 
2022-09-27T16:06:57.4347031Z ----------------------------------------------------------------------
2022-09-27T16:06:57.4347468Z Ran 1 test in 8.261s
2022-09-27T16:06:57.4347589Z 
2022-09-27T16:06:57.4347653Z OK
2022-09-27T16:06:57.4347740Z 
2022-09-27T16:06:57.4347834Z Generating XML reports...
2022-09-27T16:06:57.4394357Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160649.xml
2022-09-27T16:06:58.7486983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:58.7487378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:58.7494782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:58.7495200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:58.9008504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx9unnhr
2022-09-27T16:06:58.9009406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx9unnhr/_remote_module_non_scriptable.py
2022-09-27T16:06:59.1525667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:06:59.1535257Z 
2022-09-27T16:06:59.1535498Z Running tests...
2022-09-27T16:06:59.1536117Z ----------------------------------------------------------------------
2022-09-27T16:06:59.3869792Z   test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32038
2022-09-27T16:06:59.3896617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32039
2022-09-27T16:06:59.3923291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32040
2022-09-27T16:06:59.3951348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32041
2022-09-27T16:07:00.5266139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.5266829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.5317022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.5317682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.5433477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.5434018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.5626634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.5627225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.5678461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.5679084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.5723089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.5724025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.5792471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.5793351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.6061826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.6062458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.6892393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvs72eyn
2022-09-27T16:07:00.6893201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvs72eyn/_remote_module_non_scriptable.py
2022-09-27T16:07:00.6935816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nbujozq
2022-09-27T16:07:00.6937466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nbujozq/_remote_module_non_scriptable.py
2022-09-27T16:07:00.7052263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3b4zon5
2022-09-27T16:07:00.7053502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3b4zon5/_remote_module_non_scriptable.py
2022-09-27T16:07:00.7343986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvy4qw_b
2022-09-27T16:07:00.7345689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvy4qw_b/_remote_module_non_scriptable.py
2022-09-27T16:07:00.9400336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:00.9438608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:00.9555988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:01.0032298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:06.5081480Z ok (7.354s)
2022-09-27T16:07:06.5081769Z 
2022-09-27T16:07:06.5082152Z ----------------------------------------------------------------------
2022-09-27T16:07:06.5082407Z Ran 1 test in 7.355s
2022-09-27T16:07:06.5082538Z 
2022-09-27T16:07:06.5082600Z OK
2022-09-27T16:07:06.5082680Z 
2022-09-27T16:07:06.5082775Z Generating XML reports...
2022-09-27T16:07:06.5127410Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160659.xml
2022-09-27T16:07:07.8124707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:07.8125096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:07.8132521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:07.8132962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:07.9642968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kai03n4
2022-09-27T16:07:07.9644646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kai03n4/_remote_module_non_scriptable.py
2022-09-27T16:07:08.2156610Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:08.2167428Z 
2022-09-27T16:07:08.2167546Z Running tests...
2022-09-27T16:07:08.2167992Z ----------------------------------------------------------------------
2022-09-27T16:07:08.4534824Z   test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32213
2022-09-27T16:07:08.4562106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32214
2022-09-27T16:07:08.4589641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32215
2022-09-27T16:07:08.4617294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32216
2022-09-27T16:07:09.5778582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:09.5779543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:09.5876481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:09.5877030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:09.6086978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:09.6087660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:09.6114317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:09.6114994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:09.6131430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:09.6132068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:09.6238075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:09.6239170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:09.6450014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:09.6450674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:09.6472719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:09.6473403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:09.7392879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cd8_iqf
2022-09-27T16:07:09.7394367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cd8_iqf/_remote_module_non_scriptable.py
2022-09-27T16:07:09.7482879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx_rto4i
2022-09-27T16:07:09.7484817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx_rto4i/_remote_module_non_scriptable.py
2022-09-27T16:07:09.7712638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41dwgwz2
2022-09-27T16:07:09.7714094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41dwgwz2/_remote_module_non_scriptable.py
2022-09-27T16:07:09.7731393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_idl_tet
2022-09-27T16:07:09.7733707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_idl_tet/_remote_module_non_scriptable.py
2022-09-27T16:07:09.9915376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:09.9985741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:10.0203707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:10.0424398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:12.0944189Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-09-27T16:07:12.0945363Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2022-09-27T16:07:12.0946922Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fc8fbb1634b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:07:12.0948216Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fc8fbb11d1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:07:12.0949689Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7fc8fbb13abe in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:07:12.0950936Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7fc9002f4330 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0952600Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5cc (0x7fc9002e432c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0954241Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7fc90831dbaa in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:07:12.0955823Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7fc9002e8c1c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0957471Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fc908318c45 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:07:12.0958648Z frame #8: <unknown function> + 0x4789fa3 (0x7fc9002e1fa3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0959911Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fc9002e2b38 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0961317Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fc9002dd047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0962312Z frame #11: <unknown function> + 0x47b9a52 (0x7fc900311a52 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:07:12.0963180Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fc8fbb0477b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:07:12.0963798Z frame #13: <unknown function> + 0xc9039 (0x7fc8fba1b039 in /opt/conda/lib/libstdc++.so.6)
2022-09-27T16:07:12.0964458Z frame #14: <unknown function> + 0x8609 (0x7fc9101a8609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-09-27T16:07:12.0965085Z frame #15: clone + 0x43 (0x7fc9100cd133 in /lib/x86_64-linux-gnu/libc.so.6)
2022-09-27T16:07:12.0965351Z 
2022-09-27T16:07:12.4700909Z ok (4.253s)
2022-09-27T16:07:12.4701089Z 
2022-09-27T16:07:12.4701526Z ----------------------------------------------------------------------
2022-09-27T16:07:12.4702792Z Ran 1 test in 4.253s
2022-09-27T16:07:12.4702976Z 
2022-09-27T16:07:12.4703041Z OK
2022-09-27T16:07:12.4703144Z 
2022-09-27T16:07:12.4703520Z Generating XML reports...
2022-09-27T16:07:12.4747979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160708.xml
2022-09-27T16:07:13.7902390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:13.7903028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:13.7911688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:13.7912334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:13.9431486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp1voik5
2022-09-27T16:07:13.9432181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp1voik5/_remote_module_non_scriptable.py
2022-09-27T16:07:14.1950530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:14.1960500Z 
2022-09-27T16:07:14.1960639Z Running tests...
2022-09-27T16:07:14.1961246Z ----------------------------------------------------------------------
2022-09-27T16:07:14.4372185Z   test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32388
2022-09-27T16:07:14.4398925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32389
2022-09-27T16:07:14.4426283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32390
2022-09-27T16:07:14.4454922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32391
2022-09-27T16:07:15.5440770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.5441414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.5769828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.5770717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.5798312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.5798956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.5937650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.5938175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.6080404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.6081024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.6153060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.6153669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.6299152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.6299692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.6421334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.6421984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.7034026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xwdtb8f
2022-09-27T16:07:15.7034494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xwdtb8f/_remote_module_non_scriptable.py
2022-09-27T16:07:15.7591577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_crt5yt5
2022-09-27T16:07:15.7592540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_crt5yt5/_remote_module_non_scriptable.py
2022-09-27T16:07:15.7712363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1a_58wc
2022-09-27T16:07:15.7713746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1a_58wc/_remote_module_non_scriptable.py
2022-09-27T16:07:15.7774417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzhd7v1v
2022-09-27T16:07:15.7776562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzhd7v1v/_remote_module_non_scriptable.py
2022-09-27T16:07:15.9697985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:16.0097189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:16.0219310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:16.0455714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:29.9713488Z ok (15.775s)
2022-09-27T16:07:29.9713667Z 
2022-09-27T16:07:29.9714165Z ----------------------------------------------------------------------
2022-09-27T16:07:29.9714604Z Ran 1 test in 15.775s
2022-09-27T16:07:29.9714725Z 
2022-09-27T16:07:29.9714774Z OK
2022-09-27T16:07:29.9714866Z 
2022-09-27T16:07:29.9714959Z Generating XML reports...
2022-09-27T16:07:29.9759687Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160714.xml
2022-09-27T16:07:31.2867804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:31.2868189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:31.2876050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:31.2876424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:31.4400749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg8p0fyo
2022-09-27T16:07:31.4401892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg8p0fyo/_remote_module_non_scriptable.py
2022-09-27T16:07:31.6921626Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:31.6932699Z 
2022-09-27T16:07:31.6933179Z Running tests...
2022-09-27T16:07:31.6933602Z ----------------------------------------------------------------------
2022-09-27T16:07:31.9314736Z   test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32563
2022-09-27T16:07:31.9342152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32564
2022-09-27T16:07:31.9368819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32565
2022-09-27T16:07:31.9396924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32566
2022-09-27T16:07:32.9933456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:32.9934072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:33.0276489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:33.0277113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:33.0360368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:33.0361034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:33.0693230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:33.0694294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:33.1236924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:33.1237888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:33.1390252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:33.1390915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:33.1542755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30rz_m7g
2022-09-27T16:07:33.1543614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30rz_m7g/_remote_module_non_scriptable.py
2022-09-27T16:07:33.1580432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:33.1581105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:33.1742048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:33.1742722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:33.1950132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3sz0f3d
2022-09-27T16:07:33.1951280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3sz0f3d/_remote_module_non_scriptable.py
2022-09-27T16:07:33.2867129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo4k4zgt
2022-09-27T16:07:33.2868475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo4k4zgt/_remote_module_non_scriptable.py
2022-09-27T16:07:33.2984404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3zm7qsz
2022-09-27T16:07:33.2985440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3zm7qsz/_remote_module_non_scriptable.py
2022-09-27T16:07:33.4065773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:33.4445468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:33.5492593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:33.5570503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:41.5565223Z ok (9.863s)
2022-09-27T16:07:41.5565529Z 
2022-09-27T16:07:41.5566040Z ----------------------------------------------------------------------
2022-09-27T16:07:41.5566297Z Ran 1 test in 9.863s
2022-09-27T16:07:41.5566418Z 
2022-09-27T16:07:41.5566482Z OK
2022-09-27T16:07:41.5566946Z 
2022-09-27T16:07:41.5567052Z Generating XML reports...
2022-09-27T16:07:41.5612263Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160731.xml
2022-09-27T16:07:42.8862149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:42.8862541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:42.8870682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:42.8871035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:43.0397643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyr27u4p
2022-09-27T16:07:43.0398674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyr27u4p/_remote_module_non_scriptable.py
2022-09-27T16:07:43.2910513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:43.2920912Z 
2022-09-27T16:07:43.2921028Z Running tests...
2022-09-27T16:07:43.2921527Z ----------------------------------------------------------------------
2022-09-27T16:07:43.5255460Z   test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32738
2022-09-27T16:07:43.5281920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32739
2022-09-27T16:07:43.5308721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32740
2022-09-27T16:07:43.5337218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32741
2022-09-27T16:07:44.6534507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:44.6535147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:44.6777823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:44.6778478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:44.6879089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:44.6879692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:44.7083608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:44.7084253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:44.7140908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:44.7141555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:44.7149883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:44.7150392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:44.7444478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:44.7445104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:44.7503349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:44.7504174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:44.8165823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2o5az67
2022-09-27T16:07:44.8167333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2o5az67/_remote_module_non_scriptable.py
2022-09-27T16:07:44.8434030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex702jzf
2022-09-27T16:07:44.8435218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex702jzf/_remote_module_non_scriptable.py
2022-09-27T16:07:44.8713036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy8rm_6_
2022-09-27T16:07:44.8714692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy8rm_6_/_remote_module_non_scriptable.py
2022-09-27T16:07:44.8755868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbklq9uar
2022-09-27T16:07:44.8757028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbklq9uar/_remote_module_non_scriptable.py
2022-09-27T16:07:45.0736750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:45.1130675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:45.1251569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:45.1290666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:47.5421454Z ok (4.250s)
2022-09-27T16:07:47.5421721Z 
2022-09-27T16:07:47.5422115Z ----------------------------------------------------------------------
2022-09-27T16:07:47.5422623Z Ran 1 test in 4.250s
2022-09-27T16:07:47.5422747Z 
2022-09-27T16:07:47.5422808Z OK
2022-09-27T16:07:47.5422902Z 
2022-09-27T16:07:47.5422996Z Generating XML reports...
2022-09-27T16:07:47.5467060Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160743.xml
2022-09-27T16:07:48.8537240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:48.8537650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:48.8545337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:48.8545707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:49.0068920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpnwd4hl
2022-09-27T16:07:49.0070044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpnwd4hl/_remote_module_non_scriptable.py
2022-09-27T16:07:49.2599954Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:49.2610749Z 
2022-09-27T16:07:49.2611212Z Running tests...
2022-09-27T16:07:49.2611632Z ----------------------------------------------------------------------
2022-09-27T16:07:49.4930042Z   test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 445
2022-09-27T16:07:49.4956829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 446
2022-09-27T16:07:49.4984107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 447
2022-09-27T16:07:49.5012587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 448
2022-09-27T16:07:50.6114429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:50.6115107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:50.6119220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:50.6119880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:50.6178942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:50.6179468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:50.6476055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:50.6476803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:50.6484652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:50.6485316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:50.6541129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:50.6541783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:50.7020949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:50.7021324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:50.7368769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:50.7369410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:50.7790093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4xuh06g
2022-09-27T16:07:50.7791716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn8oadc7
2022-09-27T16:07:50.7792436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4xuh06g/_remote_module_non_scriptable.py
2022-09-27T16:07:50.7793166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn8oadc7/_remote_module_non_scriptable.py
2022-09-27T16:07:50.7822788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3ezbsfn
2022-09-27T16:07:50.7824331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3ezbsfn/_remote_module_non_scriptable.py
2022-09-27T16:07:50.8647936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqsnsykf
2022-09-27T16:07:50.8648759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqsnsykf/_remote_module_non_scriptable.py
2022-09-27T16:07:51.0312224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:51.0315893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:07:51.0318335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:07:51.1335855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:57.6156736Z ok (8.354s)
2022-09-27T16:07:57.6157000Z 
2022-09-27T16:07:57.6157531Z ----------------------------------------------------------------------
2022-09-27T16:07:57.6157840Z Ran 1 test in 8.355s
2022-09-27T16:07:57.6158034Z 
2022-09-27T16:07:57.6158161Z OK
2022-09-27T16:07:57.6158288Z 
2022-09-27T16:07:57.6158382Z Generating XML reports...
2022-09-27T16:07:57.6205323Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160749.xml
2022-09-27T16:07:58.9193857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:58.9194243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:58.9202120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:58.9202498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:59.0723637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy60egntd
2022-09-27T16:07:59.0724329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy60egntd/_remote_module_non_scriptable.py
2022-09-27T16:07:59.3224960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:07:59.3235431Z 
2022-09-27T16:07:59.3235575Z Running tests...
2022-09-27T16:07:59.3235990Z ----------------------------------------------------------------------
2022-09-27T16:07:59.5551967Z   test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 620
2022-09-27T16:07:59.5577621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 621
2022-09-27T16:07:59.5606201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 622
2022-09-27T16:07:59.5633425Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 623
2022-09-27T16:08:00.6640271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:00.6640938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:00.6713350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:00.6714017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:00.6825249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:00.6825842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:00.6910922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:00.6911473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:00.7005671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:00.7006274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:00.7082775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:00.7083306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:00.7197832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:00.7198479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:00.7274005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:00.7274646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:00.8262911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cqsvz_z
2022-09-27T16:08:00.8263701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cqsvz_z/_remote_module_non_scriptable.py
2022-09-27T16:08:00.8337362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr72kupsz
2022-09-27T16:08:00.8338769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr72kupsz/_remote_module_non_scriptable.py
2022-09-27T16:08:00.8446741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5uspnlu
2022-09-27T16:08:00.8448586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5uspnlu/_remote_module_non_scriptable.py
2022-09-27T16:08:00.8558878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuqu0hql
2022-09-27T16:08:00.8560057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuqu0hql/_remote_module_non_scriptable.py
2022-09-27T16:08:01.0782220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:01.0867801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:01.0976647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:01.1270032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:04.1804819Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-09-27T16:08:04.1805769Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2022-09-27T16:08:04.1806601Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f87f3dff34b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:08:04.1807369Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f87f3dfad1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:08:04.1808196Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f87f3dfcabe in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:08:04.1809199Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x440 (0x7f87f85dd330 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1810265Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5cc (0x7f87f85cd32c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1811258Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xcc (0x7f880060253c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:08:04.1812206Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f87f85d1c64 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1813177Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f8800601c45 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:08:04.1813857Z frame #8: <unknown function> + 0x4789fa3 (0x7f87f85cafa3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1814580Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f87f85cbb38 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1815402Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f87f85c6047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1816004Z frame #11: <unknown function> + 0x47b9a52 (0x7f87f85faa52 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:08:04.1816511Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f87f3ded77b in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-09-27T16:08:04.1816885Z frame #13: <unknown function> + 0xc9039 (0x7f87f3d04039 in /opt/conda/lib/libstdc++.so.6)
2022-09-27T16:08:04.1817297Z frame #14: <unknown function> + 0x8609 (0x7f8808491609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-09-27T16:08:04.1817670Z frame #15: clone + 0x43 (0x7f88083b6133 in /lib/x86_64-linux-gnu/libc.so.6)
2022-09-27T16:08:04.1817832Z 
2022-09-27T16:08:04.1818114Z [W tensorpipe_agent.cpp:625] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down
2022-09-27T16:08:04.4729656Z ok (5.149s)
2022-09-27T16:08:04.4729833Z 
2022-09-27T16:08:04.4730201Z ----------------------------------------------------------------------
2022-09-27T16:08:04.4730497Z Ran 1 test in 5.149s
2022-09-27T16:08:04.4730614Z 
2022-09-27T16:08:04.4730677Z OK
2022-09-27T16:08:04.4730769Z 
2022-09-27T16:08:04.4730864Z Generating XML reports...
2022-09-27T16:08:04.4776535Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160759.xml
2022-09-27T16:08:05.7835426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:05.7835807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:05.7843669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:05.7844297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:05.9366609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0z2q8c0
2022-09-27T16:08:05.9367430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0z2q8c0/_remote_module_non_scriptable.py
2022-09-27T16:08:06.1869868Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:06.1880190Z 
2022-09-27T16:08:06.1880433Z Running tests...
2022-09-27T16:08:06.1881054Z ----------------------------------------------------------------------
2022-09-27T16:08:06.4205251Z   test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 796
2022-09-27T16:08:06.4231300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 797
2022-09-27T16:08:06.4258308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 798
2022-09-27T16:08:06.4286499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 799
2022-09-27T16:08:07.5529172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:07.5529808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:07.5729982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:07.5730583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:07.5834050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:07.5834640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:07.5869108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:07.5869764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:07.6091952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:07.6092623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:07.6189831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:07.6190443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:07.6195846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:07.6196468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:07.6520019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:07.6520680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:07.7133507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30takfrw
2022-09-27T16:08:07.7134304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30takfrw/_remote_module_non_scriptable.py
2022-09-27T16:08:07.7353160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunrksa5s
2022-09-27T16:08:07.7353936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunrksa5s/_remote_module_non_scriptable.py
2022-09-27T16:08:07.7446874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc0ll35f
2022-09-27T16:08:07.7448122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc0ll35f/_remote_module_non_scriptable.py
2022-09-27T16:08:07.7793262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzcv9c40
2022-09-27T16:08:07.7796305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzcv9c40/_remote_module_non_scriptable.py
2022-09-27T16:08:07.9639955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:07.9859213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:07.9950178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:08.0479298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:14.4430916Z ok (8.255s)
2022-09-27T16:08:14.4431154Z 
2022-09-27T16:08:14.4431589Z ----------------------------------------------------------------------
2022-09-27T16:08:14.4431916Z Ran 1 test in 8.255s
2022-09-27T16:08:14.4432035Z 
2022-09-27T16:08:14.4432101Z OK
2022-09-27T16:08:14.4432192Z 
2022-09-27T16:08:14.4432289Z Generating XML reports...
2022-09-27T16:08:14.4478775Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160806.xml
2022-09-27T16:08:15.7786656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:15.7787041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:15.7795782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:15.7796131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:15.9353690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdgf0grz
2022-09-27T16:08:15.9354858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdgf0grz/_remote_module_non_scriptable.py
2022-09-27T16:08:16.1875420Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:16.1885942Z 
2022-09-27T16:08:16.1886356Z Running tests...
2022-09-27T16:08:16.1886989Z ----------------------------------------------------------------------
2022-09-27T16:08:16.4232165Z   test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 971
2022-09-27T16:08:16.4258061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 972
2022-09-27T16:08:16.4285522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 973
2022-09-27T16:08:16.4313724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 974
2022-09-27T16:08:17.5323620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:17.5324267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:17.5640951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:17.5641640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:17.5994654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:17.5995293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:17.6002362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:17.6002990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:17.6027186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:17.6027813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:17.6361449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:17.6362193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:17.6369019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:17.6369673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:17.6388699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:17.6389373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:17.6893225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fbxhbix
2022-09-27T16:08:17.6893925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fbxhbix/_remote_module_non_scriptable.py
2022-09-27T16:08:17.7628726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvwpm6pm
2022-09-27T16:08:17.7629742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvwpm6pm/_remote_module_non_scriptable.py
2022-09-27T16:08:17.7638611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhft3rpp
2022-09-27T16:08:17.7640161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhft3rpp/_remote_module_non_scriptable.py
2022-09-27T16:08:17.7649578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuinp4qo
2022-09-27T16:08:17.7650844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuinp4qo/_remote_module_non_scriptable.py
2022-09-27T16:08:17.9390104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:18.0155729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:18.0158152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:18.0318639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:24.4458504Z ok (8.257s)
2022-09-27T16:08:24.4484515Z 
2022-09-27T16:08:24.4485113Z ----------------------------------------------------------------------
2022-09-27T16:08:24.4485469Z Ran 1 test in 8.257s
2022-09-27T16:08:24.4485599Z 
2022-09-27T16:08:24.4485662Z OK
2022-09-27T16:08:24.4485754Z 
2022-09-27T16:08:24.4485860Z Generating XML reports...
2022-09-27T16:08:24.4513240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160816.xml
2022-09-27T16:08:25.7734686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:25.7735069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:25.7742620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:25.7743128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:25.9267273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0q7aanx6
2022-09-27T16:08:25.9268414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0q7aanx6/_remote_module_non_scriptable.py
2022-09-27T16:08:26.1774668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:26.1785170Z 
2022-09-27T16:08:26.1785303Z Running tests...
2022-09-27T16:08:26.1785904Z ----------------------------------------------------------------------
2022-09-27T16:08:26.4123176Z   test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1146
2022-09-27T16:08:26.4149349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1147
2022-09-27T16:08:26.4175979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1148
2022-09-27T16:08:26.4204902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1149
2022-09-27T16:08:27.5060418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:27.5061077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:27.5338158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:27.5339173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:27.5392699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:27.5393357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:27.5687501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:27.5688166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:27.5965247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:27.5965957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:27.6048616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:27.6049262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:27.6316793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:27.6317495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:27.6400387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:27.6401015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:27.6692823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfchk3nmc
2022-09-27T16:08:27.6693568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfchk3nmc/_remote_module_non_scriptable.py
2022-09-27T16:08:27.6989526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vy3s6wi
2022-09-27T16:08:27.6990353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vy3s6wi/_remote_module_non_scriptable.py
2022-09-27T16:08:27.7564844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk_h4s8x
2022-09-27T16:08:27.7565641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk_h4s8x/_remote_module_non_scriptable.py
2022-09-27T16:08:27.7677635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraael3mp
2022-09-27T16:08:27.7679002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraael3mp/_remote_module_non_scriptable.py
2022-09-27T16:08:27.9204365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:27.9498828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:28.0067910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:28.0341002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:34.4350177Z ok (8.256s)
2022-09-27T16:08:34.4350407Z 
2022-09-27T16:08:34.4350752Z ----------------------------------------------------------------------
2022-09-27T16:08:34.4351023Z Ran 1 test in 8.256s
2022-09-27T16:08:34.4351141Z 
2022-09-27T16:08:34.4351213Z OK
2022-09-27T16:08:34.4351348Z 
2022-09-27T16:08:34.4351445Z Generating XML reports...
2022-09-27T16:08:34.4396755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160826.xml
2022-09-27T16:08:35.7277131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:35.7277772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:35.7286064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:35.7286705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:35.8814220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wimi0l0
2022-09-27T16:08:35.8814923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wimi0l0/_remote_module_non_scriptable.py
2022-09-27T16:08:36.1334243Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:36.1345144Z 
2022-09-27T16:08:36.1345571Z Running tests...
2022-09-27T16:08:36.1346237Z ----------------------------------------------------------------------
2022-09-27T16:08:36.3703385Z   test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1321
2022-09-27T16:08:36.3729806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1322
2022-09-27T16:08:36.3757393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1323
2022-09-27T16:08:36.3786006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1324
2022-09-27T16:08:37.4895315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.4895955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.5231616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.5232258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.5238484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.5239063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.5443461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.5444138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.5581810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.5582354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.5792987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.5793637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.5818915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.5819548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.6144466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.6145167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.6491736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzql17wmm
2022-09-27T16:08:37.6492447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzql17wmm/_remote_module_non_scriptable.py
2022-09-27T16:08:37.6873478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h73diq7
2022-09-27T16:08:37.6874898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h73diq7/_remote_module_non_scriptable.py
2022-09-27T16:08:37.7044637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr_swc90
2022-09-27T16:08:37.7045399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr_swc90/_remote_module_non_scriptable.py
2022-09-27T16:08:37.7392935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy79rhpb9
2022-09-27T16:08:37.7393910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy79rhpb9/_remote_module_non_scriptable.py
2022-09-27T16:08:37.9013084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:37.9535069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:37.9557942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:37.9900881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:44.3928044Z ok (8.258s)
2022-09-27T16:08:44.3928352Z 
2022-09-27T16:08:44.3928725Z ----------------------------------------------------------------------
2022-09-27T16:08:44.3928983Z Ran 1 test in 8.258s
2022-09-27T16:08:44.3929101Z 
2022-09-27T16:08:44.3929150Z OK
2022-09-27T16:08:44.3929246Z 
2022-09-27T16:08:44.3929342Z Generating XML reports...
2022-09-27T16:08:44.3975410Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160836.xml
2022-09-27T16:08:45.6986925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:45.6987355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:45.6994796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:45.6995162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:45.8513815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5eiv4gb8
2022-09-27T16:08:45.8514714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5eiv4gb8/_remote_module_non_scriptable.py
2022-09-27T16:08:46.1011513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:46.1021780Z 
2022-09-27T16:08:46.1021939Z Running tests...
2022-09-27T16:08:46.1022332Z ----------------------------------------------------------------------
2022-09-27T16:08:46.3364049Z   test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1496
2022-09-27T16:08:46.3390018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1497
2022-09-27T16:08:46.3416332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1498
2022-09-27T16:08:46.3445656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1499
2022-09-27T16:08:47.4150438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:47.4151125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:47.4464773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:47.4465349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:47.4661740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:47.4662382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:47.4694463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:47.4695188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:47.4862564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:47.4863203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:47.5021676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:47.5022588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:47.5055661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:47.5056521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:47.5208223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:47.5208847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:47.5729495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a914k3w
2022-09-27T16:08:47.5730277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a914k3w/_remote_module_non_scriptable.py
2022-09-27T16:08:47.6268595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3iu7aqi
2022-09-27T16:08:47.6269374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3iu7aqi/_remote_module_non_scriptable.py
2022-09-27T16:08:47.6306359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5dzrqa0
2022-09-27T16:08:47.6307589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5dzrqa0/_remote_module_non_scriptable.py
2022-09-27T16:08:47.6490458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu27shcp9
2022-09-27T16:08:47.6491667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu27shcp9/_remote_module_non_scriptable.py
2022-09-27T16:08:47.8246649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:47.8872685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:47.8946846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:47.9197734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:54.3587940Z ok (8.256s)
2022-09-27T16:08:54.3588153Z 
2022-09-27T16:08:54.3588653Z ----------------------------------------------------------------------
2022-09-27T16:08:54.3589124Z Ran 1 test in 8.257s
2022-09-27T16:08:54.3589325Z 
2022-09-27T16:08:54.3589444Z OK
2022-09-27T16:08:54.3589609Z 
2022-09-27T16:08:54.3589778Z Generating XML reports...
2022-09-27T16:08:54.3635207Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160846.xml
2022-09-27T16:08:55.6755403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:55.6755838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:55.6763117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:55.6763491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:55.8290270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlr104oe
2022-09-27T16:08:55.8292302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlr104oe/_remote_module_non_scriptable.py
2022-09-27T16:08:56.0795682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:08:56.0805686Z 
2022-09-27T16:08:56.0805824Z Running tests...
2022-09-27T16:08:56.0806443Z ----------------------------------------------------------------------
2022-09-27T16:08:56.3241333Z   test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1671
2022-09-27T16:08:56.3267113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1672
2022-09-27T16:08:56.3294073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1673
2022-09-27T16:08:56.3322957Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1674
2022-09-27T16:08:57.4808387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.4809308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.4965555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.4966196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.5150793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.5151413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.5243454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.5244110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.5256471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.5257116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.5319850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.5320454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.5599796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.5600464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.5614039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.5614676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.6417560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cz_n9wu
2022-09-27T16:08:57.6418339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cz_n9wu/_remote_module_non_scriptable.py
2022-09-27T16:08:57.6601815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpls7rbwjo
2022-09-27T16:08:57.6602893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpls7rbwjo/_remote_module_non_scriptable.py
2022-09-27T16:08:57.6848788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l5klwug
2022-09-27T16:08:57.6849796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l5klwug/_remote_module_non_scriptable.py
2022-09-27T16:08:57.6855331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzgv456x
2022-09-27T16:08:57.6857645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzgv456x/_remote_module_non_scriptable.py
2022-09-27T16:08:57.8969483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:57.9270176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:08:57.9348398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:08:57.9371706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:04.3466457Z ok (8.266s)
2022-09-27T16:09:04.3466933Z 
2022-09-27T16:09:04.3467589Z ----------------------------------------------------------------------
2022-09-27T16:09:04.3468032Z Ran 1 test in 8.266s
2022-09-27T16:09:04.3468226Z 
2022-09-27T16:09:04.3468336Z OK
2022-09-27T16:09:04.3468473Z 
2022-09-27T16:09:04.3468613Z Generating XML reports...
2022-09-27T16:09:04.3515264Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160856.xml
2022-09-27T16:09:05.6546476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:05.6546875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:05.6554416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:05.6554778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:05.8085078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6oqt4cq
2022-09-27T16:09:05.8086068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6oqt4cq/_remote_module_non_scriptable.py
2022-09-27T16:09:06.0580725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:09:06.0591357Z 
2022-09-27T16:09:06.0591645Z Running tests...
2022-09-27T16:09:06.0592273Z ----------------------------------------------------------------------
2022-09-27T16:09:06.2922785Z   test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1846
2022-09-27T16:09:06.2949206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1847
2022-09-27T16:09:06.2976316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1848
2022-09-27T16:09:06.3005120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1849
2022-09-27T16:09:07.4240526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:07.4241122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:07.4326955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:07.4327512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:07.4446126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:07.4446677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:07.4597976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:07.4598571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:07.4689973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:07.4690610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:07.4798492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:07.4799145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:07.5065771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:07.5066404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:07.5398956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:07.5399619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:07.5858016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss2hxrne
2022-09-27T16:09:07.5858809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss2hxrne/_remote_module_non_scriptable.py
2022-09-27T16:09:07.5938017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fvy7s13
2022-09-27T16:09:07.5939379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fvy7s13/_remote_module_non_scriptable.py
2022-09-27T16:09:07.6058916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvc0kl5x_
2022-09-27T16:09:07.6060315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvc0kl5x_/_remote_module_non_scriptable.py
2022-09-27T16:09:07.6678413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwbse8ol
2022-09-27T16:09:07.6679425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwbse8ol/_remote_module_non_scriptable.py
2022-09-27T16:09:07.8391806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:09:07.8578089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:07.8598670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:09:07.9380787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:10.4090430Z ok (4.350s)
2022-09-27T16:09:10.4090685Z 
2022-09-27T16:09:10.4091220Z ----------------------------------------------------------------------
2022-09-27T16:09:10.4091562Z Ran 1 test in 4.350s
2022-09-27T16:09:10.4091679Z 
2022-09-27T16:09:10.4091740Z OK
2022-09-27T16:09:10.4091834Z 
2022-09-27T16:09:10.4091928Z Generating XML reports...
2022-09-27T16:09:10.4137328Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160906.xml
2022-09-27T16:09:11.7230823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:11.7231202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:11.7239152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:11.7239505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:11.8750249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8d0bdyo
2022-09-27T16:09:11.8752352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8d0bdyo/_remote_module_non_scriptable.py
2022-09-27T16:09:12.1372995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:09:12.1382783Z 
2022-09-27T16:09:12.1383193Z Running tests...
2022-09-27T16:09:12.1383614Z ----------------------------------------------------------------------
2022-09-27T16:09:12.3711130Z   test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2021
2022-09-27T16:09:12.3736811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2022
2022-09-27T16:09:12.3764274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2023
2022-09-27T16:09:12.3792449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2024
2022-09-27T16:09:13.4993317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:13.4993931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:13.5231530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:13.5232188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:13.5311347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:13.5312038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:13.5327323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:13.5327952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:13.5595816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:13.5596761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:13.5664700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:13.5665659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:13.5666479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:13.5667096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:13.5997526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:13.5998180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:13.6587346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ica5pqx
2022-09-27T16:09:13.6588007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ica5pqx/_remote_module_non_scriptable.py
2022-09-27T16:09:13.6895174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4bl4wj6
2022-09-27T16:09:13.6896179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4bl4wj6/_remote_module_non_scriptable.py
2022-09-27T16:09:13.6923546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchq7y3qz
2022-09-27T16:09:13.6925046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchq7y3qz/_remote_module_non_scriptable.py
2022-09-27T16:09:13.7239567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n25byn5
2022-09-27T16:09:13.7240890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n25byn5/_remote_module_non_scriptable.py
2022-09-27T16:09:13.9102531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:13.9435510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:09:13.9585421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:09:13.9726573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:18.9916531Z ok (6.853s)
2022-09-27T16:09:18.9916739Z 
2022-09-27T16:09:18.9917527Z ----------------------------------------------------------------------
2022-09-27T16:09:18.9917969Z Ran 1 test in 6.853s
2022-09-27T16:09:18.9918163Z 
2022-09-27T16:09:18.9918227Z OK
2022-09-27T16:09:18.9918307Z 
2022-09-27T16:09:18.9918401Z Generating XML reports...
2022-09-27T16:09:18.9963317Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160912.xml
2022-09-27T16:09:20.3241442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:20.3242094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:20.3250522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:20.3251161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:20.4766920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nieh7ww
2022-09-27T16:09:20.4767902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nieh7ww/_remote_module_non_scriptable.py
2022-09-27T16:09:20.7268307Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-09-27T16:09:20.7278981Z 
2022-09-27T16:09:20.7279502Z Running tests...
2022-09-27T16:09:20.7279926Z ----------------------------------------------------------------------
2022-09-27T16:09:20.9618097Z   test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2196
2022-09-27T16:09:20.9644135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2197
2022-09-27T16:09:20.9670306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2198
2022-09-27T16:09:20.9698201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2199
2022-09-27T16:09:22.0846154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.0846809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.0880769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.0881363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.1086381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.1087051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.1209292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.1209888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.1255237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.1255892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.1428376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.1429028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.1537776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.1538409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.1862028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.1862697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.2472682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcijwp4t
2022-09-27T16:09:22.2473507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcijwp4t/_remote_module_non_scriptable.py
2022-09-27T16:09:22.2544385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04_q45hd
2022-09-27T16:09:22.2547164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04_q45hd/_remote_module_non_scriptable.py
2022-09-27T16:09:22.2674766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eziqcpr
2022-09-27T16:09:22.2676115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eziqcpr/_remote_module_non_scriptable.py
2022-09-27T16:09:22.3116693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqof4smb
2022-09-27T16:09:22.3118413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqof4smb/_remote_module_non_scriptable.py
2022-09-27T16:09:22.4985330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:22.5173282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:22.5677173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:09:22.6071609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:09:27.6014890Z ok (6.873s)
2022-09-27T16:09:27.6015175Z 
2022-09-27T16:09:27.6016027Z ----------------------------------------------------------------------
2022-09-27T16:09:27.6016504Z Ran 1 test in 6.874s
2022-09-27T16:09:27.6016717Z 
2022-09-27T16:09:27.6016835Z OK
2022-09-27T16:09:27.6016992Z 
2022-09-27T16:09:27.6017178Z Generating XML reports...
2022-09-27T16:09:27.6062048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160920.xml
2022-09-27T16:09:28.1176797Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-09-27 16:09:28.117246]
2022-09-27T16:09:28.1177669Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:09:28.117331]
2022-09-27T16:09:29.3159045Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-09-27T16:09:29.3175835Z 
2022-09-27T16:09:29.3175961Z Running tests...
2022-09-27T16:09:29.3176766Z ----------------------------------------------------------------------
2022-09-27T16:09:29.5511398Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2375
2022-09-27T16:09:29.5537196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2376
2022-09-27T16:09:30.6712284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:30.6712692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:30.6715864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:30.6716227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:30.7204311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:30.7204664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:30.7208648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:30.7209009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:30.7973128Z dist init r=0, world=2
2022-09-27T16:09:30.8488993Z dist init r=1, world=2
2022-09-27T16:09:30.8684920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:30.8685380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:30.8686001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:30.8686806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:30.8690303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:30.8690831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:31.1578311Z skip: Need at least 2 CUDA devices (1.840s)
2022-09-27T16:09:31.1630207Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2403
2022-09-27T16:09:31.1663412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2404
2022-09-27T16:09:32.2565936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:32.2566311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:32.2569462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:32.2569827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:32.2572878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:32.2573248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:32.2577821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:32.2578226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:32.3826377Z dist init r=1, world=2
2022-09-27T16:09:32.3826715Z dist init r=0, world=2
2022-09-27T16:09:32.4035748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:32.4036217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:32.4036847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:32.4037388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:32.4039887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:32.4040453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:32.6695188Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:32.6710335Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2022-09-27T16:09:32.6755340Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2431
2022-09-27T16:09:32.6788596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2432
2022-09-27T16:09:33.7571129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:33.7571505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:33.7574941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:33.7575330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:33.7743135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:33.7743621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:33.7748223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:33.7748643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:33.8815228Z dist init r=0, world=2
2022-09-27T16:09:33.9019113Z dist init r=1, world=2
2022-09-27T16:09:33.9127390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:33.9127965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:33.9128792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:33.9129392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:33.9133312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:33.9133871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:34.1820450Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:34.1832461Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2022-09-27T16:09:34.1876043Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2459
2022-09-27T16:09:34.1909206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2460
2022-09-27T16:09:35.3032608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:35.3033234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:35.3036303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:35.3036883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:35.3085638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:35.3086270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:35.3089949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:35.3090313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:35.4282975Z dist init r=1, world=2
2022-09-27T16:09:35.4325737Z dist init r=0, world=2
2022-09-27T16:09:35.4535240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:35.4535803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:35.4536399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:35.4536933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:35.4640461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:35.4641093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:35.6939771Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:35.6951051Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2022-09-27T16:09:35.7014435Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2487
2022-09-27T16:09:35.7047544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2488
2022-09-27T16:09:36.7671227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:36.7671920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:36.7674319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:36.7674711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:36.7840425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:36.7840803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:36.7845076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:36.7845658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:36.8915208Z dist init r=0, world=2
2022-09-27T16:09:36.9079712Z dist init r=1, world=2
2022-09-27T16:09:36.9224391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:36.9225084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:36.9225790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:36.9226323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:36.9329921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:36.9330481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:37.2079346Z skip: Need at least 2 CUDA devices (1.514s)
2022-09-27T16:09:37.2091655Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2022-09-27T16:09:37.2136438Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2515
2022-09-27T16:09:37.2167855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2516
2022-09-27T16:09:38.3548438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:38.3548813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:38.3551464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:38.3551911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:38.3591929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:38.3592391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:38.3596406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:38.3596877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:38.4795531Z dist init r=0, world=2
2022-09-27T16:09:38.4870805Z dist init r=1, world=2
2022-09-27T16:09:38.4979790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:38.4980328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:38.4981166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:38.4982050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:38.4985182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:38.4985569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:38.7202362Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:38.7261252Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2543
2022-09-27T16:09:38.7293426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2544
2022-09-27T16:09:39.7966073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:39.7966714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:39.7968203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:39.7968783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:39.8002488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:39.8003068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:39.8007625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:39.8008228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:39.9207397Z dist init r=0, world=2
2022-09-27T16:09:39.9239560Z dist init r=1, world=2
2022-09-27T16:09:39.9347621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:39.9348073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:39.9348857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:39.9349400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:39.9453388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:39.9453808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:40.2326752Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:40.2378116Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2571
2022-09-27T16:09:40.2406420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2572
2022-09-27T16:09:41.3336163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:41.3336542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:41.3339001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:41.3339399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:41.3920251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:41.3920891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:41.3923269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:41.3923898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:41.4584460Z dist init r=1, world=2
2022-09-27T16:09:41.5185691Z dist init r=0, world=2
2022-09-27T16:09:41.5296743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:41.5297322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:41.5297963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:41.5298783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:41.5302488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:41.5302860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:41.8441950Z skip: Need at least 2 CUDA devices (1.611s)
2022-09-27T16:09:41.8504714Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2599
2022-09-27T16:09:41.8537639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2600
2022-09-27T16:09:42.9256952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:42.9257367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:42.9260668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:42.9261299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:42.9277998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:42.9278573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:42.9283506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:42.9283882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:43.0508042Z dist init r=0, world=2
2022-09-27T16:09:43.0520769Z dist init r=1, world=2
2022-09-27T16:09:43.0716818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:43.0717263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:43.0717957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:43.0718569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:43.0821761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:43.0822246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:43.3568996Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:09:43.3628530Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2627
2022-09-27T16:09:43.3659587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2628
2022-09-27T16:09:44.4467401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:44.4467781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:44.4471062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:44.4471412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:44.4582748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:44.4583101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:44.4587257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:44.4587612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:44.5715465Z dist init r=0, world=2
2022-09-27T16:09:44.5858086Z dist init r=1, world=2
2022-09-27T16:09:44.5966136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:44.5966696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:44.5967564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:44.5968257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:44.6072687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:44.6073296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:44.8690753Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:44.8750709Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2655
2022-09-27T16:09:44.8783074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2656
2022-09-27T16:09:45.9665435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:45.9665828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:45.9668785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:45.9669162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:45.9898701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:45.9899540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:45.9903145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:45.9903676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:46.0913316Z dist init r=0, world=2
2022-09-27T16:09:46.1171595Z dist init r=1, world=2
2022-09-27T16:09:46.1279988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:46.1280632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:46.1289239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:46.1289789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:46.1290191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:46.1290542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:46.3813071Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:46.3866418Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2683
2022-09-27T16:09:46.3895349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2684
2022-09-27T16:09:47.4982666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:47.4983077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:47.4986853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:47.4987297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:47.5094642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:47.5094988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:47.5098959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:47.5099355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:47.6224478Z dist init r=0, world=2
2022-09-27T16:09:47.6365545Z dist init r=1, world=2
2022-09-27T16:09:47.6474081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:47.6474658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:47.6475375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:47.6475910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:47.6480491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:47.6481108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:47.8925426Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:09:47.8986164Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2711
2022-09-27T16:09:47.9018365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2712
2022-09-27T16:09:48.9964499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:48.9965110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:48.9968000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:48.9968368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:49.0065136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:49.0065495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:49.0069421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:49.0069789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:49.1225816Z dist init r=1, world=2
2022-09-27T16:09:49.1306076Z dist init r=0, world=2
2022-09-27T16:09:49.1515215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:49.1515894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:49.1516728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:49.1517543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:49.1520936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:49.1521551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:49.4048900Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:49.4111473Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2739
2022-09-27T16:09:49.4144685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2740
2022-09-27T16:09:50.4739499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.4740183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.4743799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.4744451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.4995807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.4996174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.4999865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.5000243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.5983174Z dist init r=0, world=2
2022-09-27T16:09:50.6267773Z dist init r=1, world=2
2022-09-27T16:09:50.6393089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:50.6393713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:50.6394472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:50.6395002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:50.6398646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:50.6399232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:50.9175906Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:09:50.9235314Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2767
2022-09-27T16:09:50.9267923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2768
2022-09-27T16:09:52.0144191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:52.0144587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:52.0147596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:52.0148031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:52.0199472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:52.0199973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:52.0204189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:52.0204663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:52.1396042Z dist init r=0, world=2
2022-09-27T16:09:52.1440173Z dist init r=1, world=2
2022-09-27T16:09:52.1604339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:52.1604937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:52.1605689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:52.1606214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:52.1710183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:52.1710665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:52.4297522Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:52.4356261Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2795
2022-09-27T16:09:52.4389344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2796
2022-09-27T16:09:53.5692299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:53.5694906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:53.5695736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:53.5696357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:53.5716188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:53.5716710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:53.5721167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:53.5721755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:53.6938951Z dist init r=1, world=2
2022-09-27T16:09:53.6960156Z dist init r=0, world=2
2022-09-27T16:09:53.7170186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:53.7170624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:53.7171373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:53.7171920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:53.7176130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:53.7176717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:54.0424908Z skip: Need at least 2 CUDA devices (1.612s)
2022-09-27T16:09:54.0485561Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2823
2022-09-27T16:09:54.0518439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2824
2022-09-27T16:09:55.2153056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:55.2153490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:55.2156510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:55.2156869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:55.2190271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:55.2190779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:55.2195033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:55.2195598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:55.3397753Z dist init r=0, world=2
2022-09-27T16:09:55.3471586Z dist init r=1, world=2
2022-09-27T16:09:55.3580349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:55.3580765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:55.3581384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:55.3581909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:55.3686152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:55.3686706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:55.6549552Z skip: Need at least 2 CUDA devices (1.612s)
2022-09-27T16:09:55.6599746Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2851
2022-09-27T16:09:55.6626975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2852
2022-09-27T16:09:56.7420942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:56.7421331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:56.7424998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:56.7425623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:56.7715937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:56.7716776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:56.7719699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:56.7720529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:56.8667867Z dist init r=1, world=2
2022-09-27T16:09:56.8954049Z dist init r=0, world=2
2022-09-27T16:09:56.9077195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:56.9077642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:56.9078611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:56.9079152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:56.9183336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:56.9184130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:57.1661272Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:09:57.1718232Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2879
2022-09-27T16:09:57.1750705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2880
2022-09-27T16:09:58.2810251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:58.2810626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:58.2812896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:58.2813402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:58.2911883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:58.2912363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:58.2916269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:58.2916789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:58.4054252Z dist init r=1, world=2
2022-09-27T16:09:58.4152611Z dist init r=0, world=2
2022-09-27T16:09:58.4262029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:58.4262734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:58.4263390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:58.4264040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:58.4367791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:58.4368373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:58.6781167Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:09:58.6839993Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2907
2022-09-27T16:09:58.6879771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2908
2022-09-27T16:09:59.7597446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.7598322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.7600976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.7601790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:59.7770066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.7770686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.7775113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.7775722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:59.8838739Z dist init r=0, world=2
2022-09-27T16:09:59.9012949Z dist init r=1, world=2
2022-09-27T16:09:59.9147542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:59.9148153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:59.9148837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:59.9149368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:59.9153412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:59.9153822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:00.1911406Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:00.1970018Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2935
2022-09-27T16:10:00.2002272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2936
2022-09-27T16:10:01.3028585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:01.3029261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:01.3031726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:01.3032342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:01.3087522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:01.3088100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:01.3092357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:01.3092942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:01.4272056Z dist init r=0, world=2
2022-09-27T16:10:01.4323800Z dist init r=1, world=2
2022-09-27T16:10:01.4480687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:01.4481431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:01.4482075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:01.4482610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:01.4587012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:01.4587560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:01.7033159Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:01.7091595Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2963
2022-09-27T16:10:01.7123046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2964
2022-09-27T16:10:02.7676486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:02.7677059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:02.7679019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:02.7679593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:02.8035168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:02.8035824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:02.8038332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:02.8038941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:02.8932528Z dist init r=0, world=2
2022-09-27T16:10:02.9307129Z dist init r=1, world=2
2022-09-27T16:10:02.9416281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:02.9416682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:02.9417321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:02.9417854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:02.9521252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:02.9521739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:03.2157507Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:03.2163858Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2022-09-27T16:10:03.2210072Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2991
2022-09-27T16:10:03.2242179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2992
2022-09-27T16:10:04.3119940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:04.3120340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:04.3122624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:04.3123041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:04.3163025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:04.3163384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:04.3167557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:04.3167943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:04.4374700Z dist init r=1, world=2
2022-09-27T16:10:04.4404531Z dist init r=0, world=2
2022-09-27T16:10:04.4613965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:04.4614616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:04.4615433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:04.4615977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:04.4619270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:04.4619872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:04.7273209Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:04.7337031Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3019
2022-09-27T16:10:04.7368928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3020
2022-09-27T16:10:05.8296478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:05.8297149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:05.8300506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:05.8301127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:05.8391479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:05.8392095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:05.8396181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:05.8396796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:05.9571014Z dist init r=1, world=2
2022-09-27T16:10:05.9642057Z dist init r=0, world=2
2022-09-27T16:10:05.9851281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:05.9851743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:05.9852477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:05.9853080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:05.9856911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:05.9857486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:06.2402312Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:06.2465180Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3047
2022-09-27T16:10:06.2496665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3048
2022-09-27T16:10:07.3626959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:07.3627366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:07.3629803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:07.3630157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:07.3895650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:07.3896164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:07.3900091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:07.3900673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:07.4875820Z dist init r=0, world=2
2022-09-27T16:10:07.5133279Z dist init r=1, world=2
2022-09-27T16:10:07.5241245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:07.5241725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:07.5242430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:07.5242970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:07.5347433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:07.5348009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:07.8528148Z skip: Need at least 2 CUDA devices (1.613s)
2022-09-27T16:10:07.8592518Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3075
2022-09-27T16:10:07.8626259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3076
2022-09-27T16:10:08.9953628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:08.9954012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:08.9957183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:08.9957552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:08.9977459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:08.9977809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:08.9981778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:08.9982150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.1205453Z dist init r=1, world=2
2022-09-27T16:10:09.1225453Z dist init r=0, world=2
2022-09-27T16:10:09.1433918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:09.1434507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:09.1435114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:09.1435722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:09.1439571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:09.1440119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:09.3660402Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:09.3722936Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3103
2022-09-27T16:10:09.3755682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3104
2022-09-27T16:10:10.4649582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:10.4649956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:10.4653065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:10.4653504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:10.4805867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:10.4806447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:10.4809954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:10.4810319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:10.5903360Z dist init r=1, world=2
2022-09-27T16:10:10.6042450Z dist init r=0, world=2
2022-09-27T16:10:10.6252153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:10.6252867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:10.6253656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:10.6254496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:10.6257640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:10.8788923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:10.8789274Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:10.8851540Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3131
2022-09-27T16:10:10.8882681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3132
2022-09-27T16:10:11.9758811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:11.9759474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:11.9763029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:11.9763683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:11.9818803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:11.9819434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:11.9822981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:11.9823615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:12.1030837Z dist init r=0, world=2
2022-09-27T16:10:12.1081996Z dist init r=1, world=2
2022-09-27T16:10:12.1189867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:12.1190594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:12.1191467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:12.1192007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:12.1296074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:12.1296660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:12.3913464Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:12.3979713Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3159
2022-09-27T16:10:12.4012517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3160
2022-09-27T16:10:13.4969364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.4969753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.4972477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.4972981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.5188400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.5188779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.5192303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.5192675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.6214516Z dist init r=0, world=2
2022-09-27T16:10:13.6458273Z dist init r=1, world=2
2022-09-27T16:10:13.6624707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:13.6625314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:13.6626014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:13.6626654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:13.6630242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:13.6630906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:13.9044076Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:13.9107753Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3187
2022-09-27T16:10:13.9139122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3188
2022-09-27T16:10:15.0036009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:15.0036408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:15.0038591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:15.0038982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:15.0237032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:15.0237429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:15.0240918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:15.0241294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:15.1420362Z dist init r=1, world=2
2022-09-27T16:10:15.1493507Z dist init r=0, world=2
2022-09-27T16:10:15.1628731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:15.1629140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:15.1629912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:15.1630652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:15.1734080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:15.1734517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:15.4171640Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:15.4236861Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3215
2022-09-27T16:10:15.4269299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3216
2022-09-27T16:10:16.4981346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:16.4981748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:16.4984417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:16.4984810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:16.4995586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:16.4995932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:16.5000276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:16.5000643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:16.6229340Z dist init r=0, world=2
2022-09-27T16:10:16.6235614Z dist init r=1, world=2
2022-09-27T16:10:16.6438031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:16.6438652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:16.6439555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:16.6440191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:16.6443713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:16.6444270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:16.9301356Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:16.9356959Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3243
2022-09-27T16:10:16.9389315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3244
2022-09-27T16:10:18.0049414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.0050046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.0053717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.0054116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:18.0489251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.0489647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.0492666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.0493049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:18.1304022Z dist init r=0, world=2
2022-09-27T16:10:18.1760567Z dist init r=1, world=2
2022-09-27T16:10:18.1916523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:18.1917029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:18.1917750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:18.1918271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:18.2021829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:18.2022259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:18.4424247Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:18.4484286Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3271
2022-09-27T16:10:18.4517872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3272
2022-09-27T16:10:19.5142695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:19.5143368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:19.5147162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:19.5147785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:19.5351257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:19.5351843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:19.5355378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:19.5356028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:19.6388947Z dist init r=1, world=2
2022-09-27T16:10:19.6621352Z dist init r=0, world=2
2022-09-27T16:10:19.6798815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:19.6799396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:19.6800016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:19.6800538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:19.6904846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:19.6905524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:19.9552581Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:19.9612459Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3299
2022-09-27T16:10:19.9644581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3300
2022-09-27T16:10:21.0470630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:21.0471347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:21.0472380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:21.0473086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:21.0699477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:21.0700479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:21.0704969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:21.0705811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:21.1730721Z dist init r=0, world=2
2022-09-27T16:10:21.1988644Z dist init r=1, world=2
2022-09-27T16:10:21.2140422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:21.2141165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:21.2141900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:21.2142438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:21.2246788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:21.2247515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:21.4679216Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:21.4738087Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3327
2022-09-27T16:10:21.4770576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3328
2022-09-27T16:10:22.5448267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:22.5448951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:22.5452282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:22.5452902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:22.5620049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:22.5620696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:22.5624247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:22.5624978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:22.6701763Z dist init r=1, world=2
2022-09-27T16:10:22.6896380Z dist init r=0, world=2
2022-09-27T16:10:22.7010750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:22.7011189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:22.7012077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:22.7012617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:22.7117502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:22.7117961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:22.9804030Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:22.9866301Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3355
2022-09-27T16:10:22.9899174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3356
2022-09-27T16:10:24.0565219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:24.0566066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:24.0573258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:24.0573847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:24.1195418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:24.1196010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:24.1198397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:24.1198976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:24.1820854Z dist init r=1, world=2
2022-09-27T16:10:24.2464850Z dist init r=0, world=2
2022-09-27T16:10:24.2633278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:24.2633958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:24.2634656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:24.2635473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:24.2639236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:24.2639809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:24.4933104Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:24.5000396Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3383
2022-09-27T16:10:24.5032058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3384
2022-09-27T16:10:25.5710993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:25.5711681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:25.5715084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:25.5715716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:25.5801545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:25.5802167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:25.5805913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:25.5806550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:25.6955960Z dist init r=1, world=2
2022-09-27T16:10:25.7041289Z dist init r=0, world=2
2022-09-27T16:10:25.7164051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:25.7164528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:25.7165153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:25.7165776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:25.7169644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:25.7170003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:26.0062910Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:10:26.0122577Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3411
2022-09-27T16:10:26.0155362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3412
2022-09-27T16:10:27.1032942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:27.1033336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:27.1036400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:27.1036976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:27.1126844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:27.1127386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:27.1131344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:27.1131722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:27.2282097Z dist init r=1, world=2
2022-09-27T16:10:27.2368819Z dist init r=0, world=2
2022-09-27T16:10:27.2490371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:27.2490861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:27.2491474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:27.2492017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:27.2596634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:27.2597261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:27.5189347Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:27.5247410Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3439
2022-09-27T16:10:27.5279853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3440
2022-09-27T16:10:28.6230070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:28.6230480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:28.6233162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:28.6233545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.6330330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:28.6330669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:28.6334833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:28.6335193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.7478534Z dist init r=0, world=2
2022-09-27T16:10:28.7570836Z dist init r=1, world=2
2022-09-27T16:10:28.7686940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:28.7687598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:28.7688324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:28.7688864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:28.7793632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:28.7794205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:29.0312790Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:29.0370933Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3467
2022-09-27T16:10:29.0403040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3468
2022-09-27T16:10:30.1397452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:30.1397873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:30.1398329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:30.1398658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:30.1401378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:30.1402166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:30.1402857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:30.1403198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:30.2649050Z dist init r=0, world=2
2022-09-27T16:10:30.2649253Z dist init r=1, world=2
2022-09-27T16:10:30.2756730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:30.2759218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:30.2759918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:30.2859019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:30.2863250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:30.2863705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:30.5436324Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:30.5485293Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3495
2022-09-27T16:10:30.5513199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3496
2022-09-27T16:10:31.6193901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:31.6194471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:31.6196816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:31.6197178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:31.6412623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:31.6412977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:31.6417648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:31.6418181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:31.7447834Z dist init r=1, world=2
2022-09-27T16:10:31.7657944Z dist init r=0, world=2
2022-09-27T16:10:31.7867026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:31.7868029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:31.7868626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:31.7869432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:31.7872576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:31.7873222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:32.0544709Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:32.0600973Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3523
2022-09-27T16:10:32.0633807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3524
2022-09-27T16:10:33.1302392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:33.1302777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:33.1305359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:33.1305887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:33.1501486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:33.1501980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:33.1505922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:33.1506315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:33.2556383Z dist init r=1, world=2
2022-09-27T16:10:33.2778417Z dist init r=0, world=2
2022-09-27T16:10:33.2966510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:33.2967013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:33.2967917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:33.2968437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:33.2971487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:33.2972090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:33.5664532Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:33.5716198Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3551
2022-09-27T16:10:33.5749505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3552
2022-09-27T16:10:34.6598712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:34.6599087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:34.6601920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:34.6602312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:34.6907030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:34.6907398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:34.6911022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:34.6911531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:34.7851086Z dist init r=0, world=2
2022-09-27T16:10:34.8140577Z dist init r=1, world=2
2022-09-27T16:10:34.8260988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:34.8261391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:34.8262004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:34.8262549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:34.8366748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:34.8367233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:35.0779703Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:35.0832359Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3579
2022-09-27T16:10:35.0865059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3580
2022-09-27T16:10:36.1560511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:36.1560956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:36.1563784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:36.1564158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:36.1970954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:36.1971357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:36.1974173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:36.1974744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:36.2817719Z dist init r=0, world=2
2022-09-27T16:10:36.3246941Z dist init r=1, world=2
2022-09-27T16:10:36.3355263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:36.3355700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:36.3356499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:36.3357034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:36.3462356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:36.3462914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:36.5899124Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:36.5954189Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3607
2022-09-27T16:10:36.5987107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3608
2022-09-27T16:10:37.6907666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.6908285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.6911002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.6911364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.6939225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.6939570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.6944025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.6944465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.8171750Z dist init r=1, world=2
2022-09-27T16:10:37.8188349Z dist init r=0, world=2
2022-09-27T16:10:37.8379626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:37.8380040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:37.8380664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:37.8381196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:37.8384959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:37.8385503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:38.1016228Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:38.1071320Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3635
2022-09-27T16:10:38.1103537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3636
2022-09-27T16:10:39.2366771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:39.2367190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:39.2370050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:39.2370419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:39.2535119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:39.2535486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:39.2539394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:39.2539776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:39.3606301Z dist init r=1, world=2
2022-09-27T16:10:39.3807171Z dist init r=0, world=2
2022-09-27T16:10:39.4016793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:39.4017342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:39.4018100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:39.4018928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:39.4022400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:39.4022913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:39.7135699Z skip: Need at least 2 CUDA devices (1.612s)
2022-09-27T16:10:39.7186825Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3663
2022-09-27T16:10:39.7218781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3664
2022-09-27T16:10:40.8157937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:40.8158609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:40.8161726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:40.8162304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:40.8197388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:40.8197781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:40.8201459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:40.8201832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:40.9409983Z dist init r=1, world=2
2022-09-27T16:10:40.9434491Z dist init r=0, world=2
2022-09-27T16:10:40.9644292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:40.9645051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:40.9645656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:40.9646193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:40.9649005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:40.9649429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:41.2248691Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:41.2300210Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3691
2022-09-27T16:10:41.2332899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3692
2022-09-27T16:10:42.3211985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:42.3212596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:42.3214873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:42.3215454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:42.3374252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:42.3374824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:42.3379384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:42.3379802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:42.4459596Z dist init r=1, world=2
2022-09-27T16:10:42.4615825Z dist init r=0, world=2
2022-09-27T16:10:42.4825572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:42.4826195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:42.4827109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:42.4827729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:42.4831346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:42.4832031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:42.7363093Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:42.7414272Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3719
2022-09-27T16:10:42.7445470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3720
2022-09-27T16:10:43.8173496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:43.8174152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:43.8176952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:43.8177520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:43.8347087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:43.8347757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:43.8352037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:43.8352661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:43.9454124Z dist init r=0, world=2
2022-09-27T16:10:43.9632641Z dist init r=1, world=2
2022-09-27T16:10:43.9762915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:43.9763698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:43.9764612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:43.9765262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:43.9769121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:43.9769758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:44.2479749Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:44.2531114Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3747
2022-09-27T16:10:44.2563650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3748
2022-09-27T16:10:45.3452588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:45.3453075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:45.3455918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:45.3456419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:45.3823688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:45.3824573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:45.3828647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:45.3829280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:45.4709589Z dist init r=1, world=2
2022-09-27T16:10:45.5062728Z dist init r=0, world=2
2022-09-27T16:10:45.5271713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:45.5272460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:45.5273120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:45.5273651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:45.5377163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:45.5377710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:45.7594725Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:45.7645813Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3775
2022-09-27T16:10:45.7678171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3776
2022-09-27T16:10:46.8406020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:46.8406568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:46.8408999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:46.8409521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:46.8819265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:46.8819938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:46.8823643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:46.8824562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:46.9689574Z dist init r=1, world=2
2022-09-27T16:10:47.0071309Z dist init r=0, world=2
2022-09-27T16:10:47.0280217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:47.0280656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:47.0281308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:47.0281860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:47.0286119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:47.0286578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:47.2707666Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:47.2757995Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3803
2022-09-27T16:10:47.2791311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3804
2022-09-27T16:10:48.3891388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.3891981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.3894749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.3895113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.3998134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.3998636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.4002872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.4003381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.5132302Z dist init r=0, world=2
2022-09-27T16:10:48.5272091Z dist init r=1, world=2
2022-09-27T16:10:48.5441783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:48.5442380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:48.5443062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:48.5443599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:48.5447013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:48.5447408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:48.7820602Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:10:48.7871695Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3831
2022-09-27T16:10:48.7904850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3832
2022-09-27T16:10:49.8912444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:49.8913102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:49.8916789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:49.8917439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:49.9305986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:49.9306631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:49.9309200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:49.9309838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:50.0162516Z dist init r=0, world=2
2022-09-27T16:10:50.0670261Z dist init r=1, world=2
2022-09-27T16:10:50.0873668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:50.0874339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:50.0874959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:50.0875495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:50.0979362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:50.0979985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:50.3936609Z skip: Need at least 2 CUDA devices (1.611s)
2022-09-27T16:10:50.3988839Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3859
2022-09-27T16:10:50.4022278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3860
2022-09-27T16:10:51.5261012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:51.5261663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:51.5262444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:51.5262962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:51.5264617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:51.5265234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:51.5265998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:51.5266602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:51.6509923Z dist init r=1, world=2
2022-09-27T16:10:51.6510311Z dist init r=0, world=2
2022-09-27T16:10:51.6718435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:51.6719046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:51.6719956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:51.6720809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:51.6723102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:51.6723506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:51.9053995Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:51.9104805Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3887
2022-09-27T16:10:51.9136772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3888
2022-09-27T16:10:53.0134974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:53.0135352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:53.0138693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:53.0139040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:53.0329980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:53.0330324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:53.0334522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:53.0334863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:53.1410802Z dist init r=0, world=2
2022-09-27T16:10:53.1722647Z dist init r=1, world=2
2022-09-27T16:10:53.1921899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:53.1922444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:53.1923227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:53.1923770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:53.2027768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:53.2028339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:53.5172534Z skip: Need at least 2 CUDA devices (1.612s)
2022-09-27T16:10:53.5225010Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3915
2022-09-27T16:10:53.5258698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3916
2022-09-27T16:10:54.6260208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:54.6260864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:54.6264860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:54.6265474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:54.6302045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:54.6302637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:54.6306882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:54.6307485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:54.7507931Z dist init r=0, world=2
2022-09-27T16:10:54.7542362Z dist init r=1, world=2
2022-09-27T16:10:54.7716990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:54.7717695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:54.7718350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:54.7718919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:54.7723094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:54.7723676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:55.0289005Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:55.0339758Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3943
2022-09-27T16:10:55.0372088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3944
2022-09-27T16:10:56.1056872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:56.1057258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:56.1060236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:56.1060953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:56.1213208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:56.1213785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:56.1218042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:56.1218602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:56.2327706Z dist init r=1, world=2
2022-09-27T16:10:56.2512736Z dist init r=0, world=2
2022-09-27T16:10:56.2636292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:56.2636863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:56.2637557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.2638086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.2742733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:56.2743124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:56.5407153Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:56.5460049Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3971
2022-09-27T16:10:56.5493017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3972
2022-09-27T16:10:57.6423567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:57.6424493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:57.6428020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:57.6428658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:57.6584189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:57.6584846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:57.6588527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:57.6589133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:57.7672412Z dist init r=0, world=2
2022-09-27T16:10:57.7815732Z dist init r=1, world=2
2022-09-27T16:10:57.7923877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:57.7924344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:57.7924961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:57.7925502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:57.8029649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:57.8030241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:58.0525261Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:58.0584751Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3999
2022-09-27T16:10:58.0615914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4000
2022-09-27T16:10:59.1256883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:59.1257471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:59.1261083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:59.1261470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:59.1438562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:59.1438996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:59.1442850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:59.1443339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:59.2506021Z dist init r=0, world=2
2022-09-27T16:10:59.2839386Z dist init r=1, world=2
2022-09-27T16:10:59.2947707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:59.2948398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:59.2949230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:59.2949827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:59.2953499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:59.5651816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:59.5652165Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:10:59.5709618Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4027
2022-09-27T16:10:59.5742343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4028
2022-09-27T16:11:00.6588935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:00.6589645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:00.6591313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:00.6592004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:00.6796558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:00.6797191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:00.6801823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:00.6802207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:00.7867907Z dist init r=1, world=2
2022-09-27T16:11:00.8034528Z dist init r=0, world=2
2022-09-27T16:11:00.8177455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:00.8177966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:00.8178625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:00.8179202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:00.8283345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:00.8284108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:01.0773346Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:11:01.0831997Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4055
2022-09-27T16:11:01.0865345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4056
2022-09-27T16:11:02.1638594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:02.1639277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:02.1641446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:02.1642085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:02.1715827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:02.1716388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:02.1720718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:02.1721290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:02.2890149Z dist init r=1, world=2
2022-09-27T16:11:02.2963547Z dist init r=0, world=2
2022-09-27T16:11:02.3099134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:02.3099560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:02.3100170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:02.3100715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:02.3216781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:02.3233107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:02.5897166Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:11:02.5956441Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4083
2022-09-27T16:11:02.5990702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4084
2022-09-27T16:11:03.6748091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:03.6748485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:03.6751414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:03.6751833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:03.6873934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:03.6874287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:03.6878450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:03.6878812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:03.7990502Z dist init r=1, world=2
2022-09-27T16:11:03.8115893Z dist init r=0, world=2
2022-09-27T16:11:03.8325425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:03.8326373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:03.8327007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:03.8327610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:03.8334057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:03.8334688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:04.1021630Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:11:04.1079202Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4111
2022-09-27T16:11:04.1111727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4112
2022-09-27T16:11:05.1804986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:05.1805458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:05.1807410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:05.1807951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:05.1917798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:05.1918270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:05.1922159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:05.1922667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:05.3047328Z dist init r=0, world=2
2022-09-27T16:11:05.3192245Z dist init r=1, world=2
2022-09-27T16:11:05.3356645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:05.3357331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:05.3357961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:05.3358765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:05.3362319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:05.3362909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:05.6146872Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:11:05.6214269Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4139
2022-09-27T16:11:05.6246798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4140
2022-09-27T16:11:06.6871342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:06.6872001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:06.6875843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:06.6876464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:06.7144983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:06.7145588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:06.7148589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:06.7149217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:06.8125421Z dist init r=0, world=2
2022-09-27T16:11:06.8408949Z dist init r=1, world=2
2022-09-27T16:11:06.8535056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:06.8535522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:06.8536357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:06.8536981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:06.8540985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:06.8541414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:07.1277537Z skip: Need at least 2 CUDA devices (1.513s)
2022-09-27T16:11:07.1332444Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4167
2022-09-27T16:11:07.1366046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4168
2022-09-27T16:11:08.2497080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:08.2497478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:08.2500821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:08.2501250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:08.2637896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:08.2638291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:08.2642229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:08.2642600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:08.3750454Z dist init r=1, world=2
2022-09-27T16:11:08.3916832Z dist init r=0, world=2
2022-09-27T16:11:08.4059433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:08.4059841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:08.4060628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:08.4061152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:08.4166293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:08.4166989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:08.7398177Z skip: Need at least 2 CUDA devices (1.612s)
2022-09-27T16:11:08.7451895Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4195
2022-09-27T16:11:09.7997057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:09.7997479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:09.7999926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:09.8000436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:09.9232970Z dist init r=0, world=1
2022-09-27T16:11:09.9240425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:09.9241138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:09.9244673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:10.1481882Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:10.1533142Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4209
2022-09-27T16:11:11.2359895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:11.2360285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:11.2362032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:11.2362391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:11.3603347Z dist init r=0, world=1
2022-09-27T16:11:11.3611881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:11.3612575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:11.3616206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:11.5562248Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:11.5605576Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4223
2022-09-27T16:11:12.6142465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.6142853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.6145301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.6145724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.7383709Z dist init r=0, world=1
2022-09-27T16:11:12.7392845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:12.7393638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:12.7397065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:12.9637184Z skip: Need at least 2 CUDA devices (1.407s)
2022-09-27T16:11:12.9687562Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4237
2022-09-27T16:11:14.0291037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:14.0291416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:14.0293921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:14.0294267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:14.1525999Z dist init r=0, world=1
2022-09-27T16:11:14.1534739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:14.1536161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:14.1539323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:14.3717755Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:14.3768702Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4251
2022-09-27T16:11:15.4302684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:15.4303069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:15.4305956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:15.4306331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:15.5544123Z dist init r=0, world=1
2022-09-27T16:11:15.5552318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:15.5553399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:15.5556826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:15.7799746Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:15.7851307Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4265
2022-09-27T16:11:16.8443329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:16.8443711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:16.8446226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:16.8446647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:16.9683392Z dist init r=0, world=1
2022-09-27T16:11:16.9691886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:16.9692569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:16.9696070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:17.1881040Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:17.1933148Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4279
2022-09-27T16:11:18.2755803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:18.2756173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:18.2759341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:18.2759697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:18.3990004Z dist init r=0, world=1
2022-09-27T16:11:18.3998569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:18.3999338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:18.4003112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:18.5960088Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:18.6010745Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4293
2022-09-27T16:11:19.6498280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:19.6498876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:19.6501062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:19.6501496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:19.7737726Z dist init r=0, world=1
2022-09-27T16:11:19.7745807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:19.7746859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:11:19.7750094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:20.0038262Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:11:20.0038544Z 
2022-09-27T16:11:20.0039057Z ----------------------------------------------------------------------
2022-09-27T16:11:20.0039418Z Ran 73 tests in 110.686s
2022-09-27T16:11:20.0039538Z 
2022-09-27T16:11:20.0039616Z OK (skipped=73)
2022-09-27T16:11:20.0039726Z 
2022-09-27T16:11:20.0039812Z Generating XML reports...
2022-09-27T16:11:20.0143034Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220927160929.xml
2022-09-27T16:11:20.0153199Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220927160929.xml
2022-09-27T16:11:20.2953240Z Running distributed/fsdp/test_fsdp_core ... [2022-09-27 16:11:20.294925]
2022-09-27T16:11:20.2954100Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:11:20.295003]
2022-09-27T16:11:21.4828606Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core
2022-09-27T16:11:21.4845948Z 
2022-09-27T16:11:21.4846068Z Running tests...
2022-09-27T16:11:21.4847293Z ----------------------------------------------------------------------
2022-09-27T16:11:21.4852799Z   test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks)
2022-09-27T16:11:21.7196035Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4318
2022-09-27T16:11:21.7220926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4319
2022-09-27T16:11:21.7248419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4320
2022-09-27T16:11:21.7278131Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4321
2022-09-27T16:11:22.8985052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:22.8985653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:22.8988481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:22.8989075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:22.9120851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:22.9121460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:22.9125666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:22.9126319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:22.9231086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:22.9231671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:22.9234980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:22.9235851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:22.9887704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:22.9888338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:22.9891805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:22.9892462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:23.0275062Z dist init r=0, world=4
2022-09-27T16:11:23.0396038Z dist init r=1, world=4
2022-09-27T16:11:23.0508301Z dist init r=2, world=4
2022-09-27T16:11:23.1179676Z dist init r=3, world=4
2022-09-27T16:11:23.1321609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:23.1390586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:23.1391558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:23.1392030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:23.1392402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:23.1392918Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:23.1393444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:23.1423501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:23.1499032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:23.1499637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:23.1500193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:23.1500701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:23.4323145Z skip: Need at least 2 CUDA devices (1.947s)
2022-09-27T16:11:23.4328806Z   test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks)
2022-09-27T16:11:23.4373844Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4374
2022-09-27T16:11:23.4406182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4375
2022-09-27T16:11:23.4435012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4376
2022-09-27T16:11:23.4463037Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4377
2022-09-27T16:11:24.5959204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:24.5959907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:24.5962703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:24.5963400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:24.6211140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:24.6211761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:24.6214847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:24.6215490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:24.6220336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:24.6220961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:24.6225222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:24.6225816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:24.6438187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:24.6438754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:24.6443361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:24.6443981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:24.7226223Z dist init r=0, world=4
2022-09-27T16:11:24.7482977Z dist init r=2, world=4
2022-09-27T16:11:24.7511137Z dist init r=3, world=4
2022-09-27T16:11:24.7690886Z dist init r=1, world=4
2022-09-27T16:11:24.8037953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:24.8138947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:24.8242602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:24.8243812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:24.8244240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:24.8244748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:24.8245281Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:24.8245789Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:24.8269618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:24.8270186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:24.8270757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:24.8271169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:25.1504622Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:25.1510132Z   test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks)
2022-09-27T16:11:25.1554897Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4430
2022-09-27T16:11:25.1588088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4431
2022-09-27T16:11:25.1616099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4432
2022-09-27T16:11:25.1644102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4433
2022-09-27T16:11:26.3386244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:26.3386794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:26.3387433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:26.3388155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:26.3389218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:26.3389832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:26.3391693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:26.3392201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:26.3729254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:26.3729909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:26.3732833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:26.3733499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:26.3734291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:26.3734847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:26.3737560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:26.3738181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:26.4655534Z dist init r=2, world=4
2022-09-27T16:11:26.4657581Z dist init r=0, world=4
2022-09-27T16:11:26.4981431Z dist init r=1, world=4
2022-09-27T16:11:26.5027414Z dist init r=3, world=4
2022-09-27T16:11:26.5266334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:26.5367020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:26.5469749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:26.5470352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:26.5471384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:26.5472345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:26.5472989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:26.5473530Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:26.5477441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:26.5478004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:26.5478551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:26.5479082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:26.8685073Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:26.8694759Z   test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks)
2022-09-27T16:11:26.8740280Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4486
2022-09-27T16:11:26.8774310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4487
2022-09-27T16:11:26.8801709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4488
2022-09-27T16:11:26.8829539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4489
2022-09-27T16:11:28.0631111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.0631689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.0634618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.0635350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.0966044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.0966700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.0970071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.0970680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.1128841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.1129459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.1132405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.1133017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.1340395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.1341026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.1345031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.1345647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.1904670Z dist init r=3, world=4
2022-09-27T16:11:28.2231565Z dist init r=2, world=4
2022-09-27T16:11:28.2387157Z dist init r=0, world=4
2022-09-27T16:11:28.2633353Z dist init r=1, world=4
2022-09-27T16:11:28.2817416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:28.2998731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:28.3100326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:28.3101051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:28.3101868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:28.3102398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:28.3102936Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:28.3121058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:28.3207654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:28.3208292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:28.3208805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:28.3209199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:28.6872592Z skip: Need at least 2 CUDA devices (1.819s)
2022-09-27T16:11:28.6882794Z   test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks)
2022-09-27T16:11:28.6929089Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4542
2022-09-27T16:11:28.6963715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4543
2022-09-27T16:11:28.6992255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4544
2022-09-27T16:11:28.7020012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4545
2022-09-27T16:11:29.8376342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:29.8377029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:29.8379760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:29.8380392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:29.8546418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:29.8547032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:29.8551343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:29.8551908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:29.8599681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:29.8600182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:29.8604146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:29.8604726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:29.8904440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:29.8905140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:29.8908690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:29.8909311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:29.9655460Z dist init r=3, world=4
2022-09-27T16:11:29.9816069Z dist init r=1, world=4
2022-09-27T16:11:29.9855474Z dist init r=0, world=4
2022-09-27T16:11:30.0187947Z dist init r=2, world=4
2022-09-27T16:11:30.0366200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:30.0466968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:30.0569123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:30.0569825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:30.0571028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:30.0571712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:30.0572316Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:30.0574797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:30.0579929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:30.0580584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:30.0581218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:30.0581948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:30.4061379Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:30.4072664Z   test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks)
2022-09-27T16:11:30.4116968Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4598
2022-09-27T16:11:30.4150885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4599
2022-09-27T16:11:30.4179412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4600
2022-09-27T16:11:30.4208094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4601
2022-09-27T16:11:31.5957732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:31.5958319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:31.5961292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:31.5961910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:31.6115083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:31.6115741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:31.6119568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:31.6120200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:31.6162188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:31.6162784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:31.6166642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:31.6167331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:31.6381603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:31.6382218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:31.6386438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:31.6387057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:31.7227518Z dist init r=2, world=4
2022-09-27T16:11:31.7381717Z dist init r=3, world=4
2022-09-27T16:11:31.7430534Z dist init r=1, world=4
2022-09-27T16:11:31.7670087Z dist init r=0, world=4
2022-09-27T16:11:31.7938899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:31.8040154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:31.8142724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:31.8143691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:31.8144605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:31.8145264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:31.8145901Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:31.8146449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:31.8150734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:31.8151402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:31.8151970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:31.8152439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:32.1249584Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:32.1259233Z   test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks)
2022-09-27T16:11:32.1303191Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4654
2022-09-27T16:11:32.1340489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4655
2022-09-27T16:11:32.1368028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4656
2022-09-27T16:11:32.1403534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4657
2022-09-27T16:11:33.2955383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.2956041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.2958943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.2959554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.2997383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.2998031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.3002154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.3002764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.3122692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.3123315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.3127531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.3128117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.3306784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.3307470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.3311373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.3311987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.4217215Z dist init r=2, world=4
2022-09-27T16:11:33.4263657Z dist init r=1, world=4
2022-09-27T16:11:33.4424709Z dist init r=3, world=4
2022-09-27T16:11:33.4554065Z dist init r=0, world=4
2022-09-27T16:11:33.4734450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:33.4834941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:33.4937043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:33.4937629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:33.4938603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:33.4939600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:33.4940733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:33.4941272Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:33.5043503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:33.5044078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:33.5044637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:33.5045167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:33.8443839Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:33.8454469Z   test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad)
2022-09-27T16:11:33.8500640Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4710
2022-09-27T16:11:33.8534460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4711
2022-09-27T16:11:33.8562324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4712
2022-09-27T16:11:33.8590362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4713
2022-09-27T16:11:35.0278703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:35.0279427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:35.0280252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:35.0280823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:35.0281497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:35.0282056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:35.0282842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:35.0283456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.0284191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:35.0284784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.0285559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:35.0286108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.0346791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:35.0347409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:35.0351105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:35.0351717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.1546108Z dist init r=0, world=4
2022-09-27T16:11:35.1546467Z dist init r=2, world=4
2022-09-27T16:11:35.1546785Z dist init r=3, world=4
2022-09-27T16:11:35.1631323Z dist init r=1, world=4
2022-09-27T16:11:35.1754812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:35.1942363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:35.1943128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:35.1943537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:35.1944526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:35.1945070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:35.1945674Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:35.1958030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:35.2049440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:35.2050097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:35.2050458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:35.2050819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:35.5631513Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:35.5641874Z   test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad)
2022-09-27T16:11:35.5685871Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4766
2022-09-27T16:11:35.5718872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4767
2022-09-27T16:11:35.5746435Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4768
2022-09-27T16:11:35.5775259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4769
2022-09-27T16:11:36.7361187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:36.7361755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:36.7364703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:36.7365441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:36.7428694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:36.7429312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:36.7433309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:36.7433852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:36.7568869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:36.7569478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:36.7573189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:36.7573824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:36.7621293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:36.7621911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:36.7626167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:36.7626802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:36.8632162Z dist init r=0, world=4
2022-09-27T16:11:36.8707039Z dist init r=2, world=4
2022-09-27T16:11:36.8828014Z dist init r=3, world=4
2022-09-27T16:11:36.8908023Z dist init r=1, world=4
2022-09-27T16:11:36.9036933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:36.9137519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:36.9238797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:36.9242304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:36.9243055Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:36.9340857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:36.9341514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:36.9342685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:36.9447815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:36.9448512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:36.9449021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:36.9449585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:37.2814894Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:37.2826889Z   test_param_change_after_init_mixed_precision_False (__main__.TestParamInit)
2022-09-27T16:11:37.2874220Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4822
2022-09-27T16:11:37.2909075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4823
2022-09-27T16:11:37.2937499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4824
2022-09-27T16:11:37.2965760Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4825
2022-09-27T16:11:38.4611708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.4612351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.4615403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.4616026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.4635434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.4636085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.4640780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.4641413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.4785851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.4786480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.4791004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.4791627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.4975342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.4975953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.4980170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.4980804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.5904388Z dist init r=2, world=4
2022-09-27T16:11:38.5918534Z dist init r=0, world=4
2022-09-27T16:11:38.6055204Z dist init r=1, world=4
2022-09-27T16:11:38.6222397Z dist init r=3, world=4
2022-09-27T16:11:38.6330149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:38.6417476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:38.6418175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:38.6418554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:38.6419156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:38.6419685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:38.6420213Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:38.6432924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:38.6523946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:38.6524426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:38.6524953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:38.6525479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:39.0007848Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:39.0017917Z   test_param_change_after_init_mixed_precision_True (__main__.TestParamInit)
2022-09-27T16:11:39.0062344Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4878
2022-09-27T16:11:39.0095804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4879
2022-09-27T16:11:39.0122654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4880
2022-09-27T16:11:39.0150896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4881
2022-09-27T16:11:40.1445704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:40.1446331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:40.1449219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:40.1449779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:40.1528774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:40.1529365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:40.1533657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:40.1534491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:40.1869924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:40.1870554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:40.1873712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:40.1874371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:40.2122133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:40.2122992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:40.2126529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:40.2127343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:40.2720381Z dist init r=0, world=4
2022-09-27T16:11:40.2800003Z dist init r=1, world=4
2022-09-27T16:11:40.3167562Z dist init r=3, world=4
2022-09-27T16:11:40.3381776Z dist init r=2, world=4
2022-09-27T16:11:40.3578009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:40.3614316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:40.3615172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:40.3615850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:40.3616647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:40.3617192Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:40.3617728Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:40.3679845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:40.3721942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:40.3722525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:40.3723104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:40.3723656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:40.7193597Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:40.7200710Z   test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP)
2022-09-27T16:11:40.7245935Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4934
2022-09-27T16:11:40.7278601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4935
2022-09-27T16:11:40.7307002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4936
2022-09-27T16:11:40.7336715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4937
2022-09-27T16:11:41.8794092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:41.8794725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:41.8797597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:41.8798203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:41.9056081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:41.9056606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:41.9060290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:41.9060798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:41.9383149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:41.9383814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:41.9385034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:41.9385659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:41.9386632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:41.9387404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:41.9388953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:41.9389659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:42.0065808Z dist init r=1, world=4
2022-09-27T16:11:42.0356614Z dist init r=2, world=4
2022-09-27T16:11:42.0638329Z dist init r=0, world=4
2022-09-27T16:11:42.0642033Z dist init r=3, world=4
2022-09-27T16:11:42.0851923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:42.0852335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:42.0953844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:42.0954426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:42.0955288Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:42.0956240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:42.0956888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:42.0957418Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:42.1060743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:42.1061421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:42.1061958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:42.1062500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:42.4377727Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:42.4384375Z   test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP)
2022-09-27T16:11:42.4431145Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4990
2022-09-27T16:11:42.4464624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4991
2022-09-27T16:11:42.4493385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4992
2022-09-27T16:11:42.4523290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4993
2022-09-27T16:11:43.6305237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.6305871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.6308504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.6309107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.6443643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.6444187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.6448292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.6449151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.6505328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.6506113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.6509723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.6510468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.6737694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.6738237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.6742759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.6743440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.7572352Z dist init r=2, world=4
2022-09-27T16:11:43.7710576Z dist init r=3, world=4
2022-09-27T16:11:43.7769425Z dist init r=0, world=4
2022-09-27T16:11:43.8029381Z dist init r=1, world=4
2022-09-27T16:11:43.8284011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:43.8385240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:43.8487555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:43.8489807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:43.8490418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:43.8491185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:43.8492001Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:43.8492552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:43.8495991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:43.8496676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:43.8504937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:43.8505467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:44.1564789Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:11:44.1571350Z   test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2022-09-27T16:11:44.1616784Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5046
2022-09-27T16:11:44.1648689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5047
2022-09-27T16:11:44.1676717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5048
2022-09-27T16:11:44.1704253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5049
2022-09-27T16:11:45.3335218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:45.3335858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:45.3339254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:45.3339861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:45.3402561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:45.3403169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:45.3406050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:45.3406808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:45.3425763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:45.3426366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:45.3430325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:45.3430933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:45.3782866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:45.3783518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:45.3787247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:45.3787891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:45.4613830Z dist init r=2, world=4
2022-09-27T16:11:45.4663256Z dist init r=0, world=4
2022-09-27T16:11:45.4680574Z dist init r=1, world=4
2022-09-27T16:11:45.5062721Z dist init r=3, world=4
2022-09-27T16:11:45.5170139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:45.5272288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:45.5373014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:45.5373931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:45.5374864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:45.5375605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:45.5376232Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:45.5376741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:45.5480374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:45.5480830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:45.5481358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:45.5481889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:45.8746500Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:45.8752335Z   test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP)
2022-09-27T16:11:45.8756459Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:11:45.8760960Z   test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP)
2022-09-27T16:11:45.8804161Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5102
2022-09-27T16:11:45.8837594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5103
2022-09-27T16:11:45.8865605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5104
2022-09-27T16:11:45.8893692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5105
2022-09-27T16:11:47.0459960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.0463170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.0464233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.0464942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.0615727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.0616380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.0620321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.0620959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.0758364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.0758997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.0763702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.0764798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.0805171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.0806061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.0809774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.0810372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.1729163Z dist init r=0, world=4
2022-09-27T16:11:47.1878832Z dist init r=3, world=4
2022-09-27T16:11:47.2016050Z dist init r=1, world=4
2022-09-27T16:11:47.2089937Z dist init r=2, world=4
2022-09-27T16:11:47.2299485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:47.2400723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:47.2401258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:47.2401684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:47.2402288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:47.2402841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:47.2403372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:47.2403892Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:47.2508525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:47.2509167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:47.2509959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:47.2510490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:47.5934994Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:47.5941686Z   test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2022-09-27T16:11:47.5988351Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5158
2022-09-27T16:11:47.6021486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5159
2022-09-27T16:11:47.6049525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5160
2022-09-27T16:11:47.6076639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5161
2022-09-27T16:11:48.7769157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:48.7769771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:48.7772581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:48.7773191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.7875969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:48.7876588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:48.7880728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:48.7881368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.7960498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:48.7961092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:48.7964853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:48.7965495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.8182471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:48.8183115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:48.8186743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:48.8187376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.9069798Z dist init r=1, world=4
2022-09-27T16:11:48.9232616Z dist init r=3, world=4
2022-09-27T16:11:48.9265684Z dist init r=2, world=4
2022-09-27T16:11:48.9441731Z dist init r=0, world=4
2022-09-27T16:11:48.9853124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:48.9853532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:48.9955046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:48.9955640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:48.9956560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:48.9957279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:48.9957959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:48.9958767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:48.9962158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:48.9962939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:48.9963474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:48.9964009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:49.3119102Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:49.3125745Z   test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP)
2022-09-27T16:11:49.3170321Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5214
2022-09-27T16:11:49.3203760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5215
2022-09-27T16:11:49.3232724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5216
2022-09-27T16:11:49.3261491Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5217
2022-09-27T16:11:50.4800830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:50.4801469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:50.4804061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:50.4804705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:50.5086654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:50.5087278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:50.5090528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:50.5091169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:50.5725687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:50.5726331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:50.5727316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:50.5727905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:50.5728868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:50.5729486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:50.5732599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:50.5733250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:50.6082367Z dist init r=0, world=4
2022-09-27T16:11:50.6382444Z dist init r=3, world=4
2022-09-27T16:11:50.6979940Z dist init r=2, world=4
2022-09-27T16:11:50.6982830Z dist init r=1, world=4
2022-09-27T16:11:50.7190239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:50.7297328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:50.7399828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:50.7400671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:50.7401094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:50.7401609Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:50.7402125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:50.7494061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:50.7507346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:50.7508169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:50.7508827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:50.7509515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:51.0302857Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:51.0309500Z   test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP)
2022-09-27T16:11:51.0313313Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:11:51.0317261Z   test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2022-09-27T16:11:51.0319632Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:11:51.0324035Z   test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP)
2022-09-27T16:11:51.0366750Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5270
2022-09-27T16:11:51.0400446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5271
2022-09-27T16:11:51.0429357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5272
2022-09-27T16:11:51.0456160Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5273
2022-09-27T16:11:52.2066347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:52.2066994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:52.2069007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:52.2069599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:52.2565081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:52.2565674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:52.2568987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:52.2569592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:52.2594733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:52.2595323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:52.2599019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:52.2599628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:52.2763448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:52.2764059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:52.2768382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:52.2769042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:52.3339908Z dist init r=1, world=4
2022-09-27T16:11:52.3822563Z dist init r=3, world=4
2022-09-27T16:11:52.3894273Z dist init r=0, world=4
2022-09-27T16:11:52.4017913Z dist init r=2, world=4
2022-09-27T16:11:52.4131099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:52.4232340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:52.4306114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:52.4306520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:52.4307170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:52.4307705Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:52.4335591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:52.4336319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:52.4414116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:52.4414762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:52.4415334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:52.4415673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:52.7496909Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:52.7503332Z   test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP)
2022-09-27T16:11:52.7507519Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:11:52.7512405Z   test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2022-09-27T16:11:52.7514661Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:11:52.7562427Z   test_mixture_of_experts_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5326
2022-09-27T16:11:52.7595692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5327
2022-09-27T16:11:52.7623246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5328
2022-09-27T16:11:52.7650395Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5329
2022-09-27T16:11:53.9448045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:53.9448669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:53.9451064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:53.9451916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:53.9742418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:53.9743253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:53.9744879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:53.9745456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:54.0417459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:54.0418040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:54.0420830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:54.0421489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:54.0441608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:54.0442232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:54.0446061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:54.0446671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:54.0891627Z dist init r=1, world=4
2022-09-27T16:11:54.1030073Z dist init r=0, world=4
2022-09-27T16:11:54.1683889Z dist init r=2, world=4
2022-09-27T16:11:54.1729391Z dist init r=3, world=4
2022-09-27T16:11:54.1837648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:54.1993510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:54.2094974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:54.2095572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:54.2096328Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:54.2096907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:54.2097434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:54.2142419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:54.2203809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:54.2224692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:54.2225353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:54.2225998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:54.5694172Z skip: Need at least 2 CUDA devices (1.818s)
2022-09-27T16:11:54.5746650Z   test_mixture_of_experts_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5382
2022-09-27T16:11:54.5780626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5383
2022-09-27T16:11:54.5810191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5384
2022-09-27T16:11:54.5839153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5385
2022-09-27T16:11:55.7126804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:55.7127632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:55.7129959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:55.7130718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:55.7488694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:55.7489330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:55.7493289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:55.7493967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:55.7794810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:55.7795421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:55.7797806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:55.7798484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:55.7961820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:55.7962433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:55.7965654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:55.7966305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:55.8400609Z dist init r=0, world=4
2022-09-27T16:11:55.8805184Z dist init r=1, world=4
2022-09-27T16:11:55.9048140Z dist init r=2, world=4
2022-09-27T16:11:55.9211518Z dist init r=3, world=4
2022-09-27T16:11:55.9359631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:55.9360216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:55.9360810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:55.9361334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:55.9361948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:55.9362490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:55.9363015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:55.9363596Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:55.9466155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:55.9466641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:55.9467163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:55.9467711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:56.2880078Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:56.2932142Z   test_mixture_of_experts_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5438
2022-09-27T16:11:56.2966161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5439
2022-09-27T16:11:56.2993891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5440
2022-09-27T16:11:56.3022192Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5441
2022-09-27T16:11:57.4460709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:57.4461638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:57.4463601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:57.4464400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:57.4946791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:57.4947412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:57.4950767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:57.4951412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:57.5068257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:57.5068901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:57.5072570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:57.5073230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:57.5146648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:57.5147209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:57.5151532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:57.5152592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:57.5737933Z dist init r=2, world=4
2022-09-27T16:11:57.6207771Z dist init r=1, world=4
2022-09-27T16:11:57.6333212Z dist init r=0, world=4
2022-09-27T16:11:57.6438648Z dist init r=3, world=4
2022-09-27T16:11:57.6744067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:57.6845221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:57.6948269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:57.6949600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:57.6950285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:57.6950752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:57.6951257Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:57.6951787Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:57.6955361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:57.6955905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:57.6956435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:57.6956973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:58.0063777Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:58.0116073Z   test_mixture_of_experts_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5494
2022-09-27T16:11:58.0149428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5495
2022-09-27T16:11:58.0177710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5496
2022-09-27T16:11:58.0206342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5497
2022-09-27T16:11:59.1345706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:59.1346347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:59.1348472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:59.1349043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:59.1716865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:59.1717460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:59.1720388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:59.1720981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:59.2041214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:59.2041823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:59.2044808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:59.2045452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:59.2624712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:59.2625096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:59.2627604Z dist init r=0, world=4
2022-09-27T16:11:59.2645148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:59.2645762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:59.3007189Z dist init r=1, world=4
2022-09-27T16:11:59.3474004Z dist init r=2, world=4
2022-09-27T16:11:59.3928591Z dist init r=3, world=4
2022-09-27T16:11:59.4037037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:59.4138325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:59.4239709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:11:59.4240510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:11:59.4241869Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:59.4242634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:59.4243172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:59.4243710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:11:59.4250391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:11:59.4264965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:11:59.4265578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:59.4266442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:59.7248817Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:11:59.7301332Z   test_mixture_of_experts_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5550
2022-09-27T16:11:59.7334628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5551
2022-09-27T16:11:59.7362972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5552
2022-09-27T16:11:59.7391789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5553
2022-09-27T16:12:00.8933693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:00.8934378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:00.8937601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:00.8938161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:00.8991163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:00.8991825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:00.8995716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:00.8996384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:00.9081639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:00.9082288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:00.9085600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:00.9086251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:00.9278178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:00.9278824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:00.9282441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:00.9283078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:01.0251360Z dist init r=1, world=4
2022-09-27T16:12:01.0272504Z dist init r=3, world=4
2022-09-27T16:12:01.0352003Z dist init r=0, world=4
2022-09-27T16:12:01.0548388Z dist init r=2, world=4
2022-09-27T16:12:01.0756857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:01.0763920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:01.0764898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:01.0765997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:01.0766413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:01.0766898Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:01.0767418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:01.0859828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:01.0872165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:01.0874358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:01.0874926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:01.0875571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:01.4432837Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:01.4485121Z   test_mixture_of_experts_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5606
2022-09-27T16:12:01.4517909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5607
2022-09-27T16:12:01.4546071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5608
2022-09-27T16:12:01.4573860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5609
2022-09-27T16:12:02.6285059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.6285658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.6289244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.6289894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.6434274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.6434806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.6439277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.6439909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.6447201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.6447800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.6451160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.6451799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.6494669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.6495262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.6499892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.6500512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.7587149Z dist init r=1, world=4
2022-09-27T16:12:02.7702174Z dist init r=3, world=4
2022-09-27T16:12:02.7717521Z dist init r=2, world=4
2022-09-27T16:12:02.7751481Z dist init r=0, world=4
2022-09-27T16:12:02.8028336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:02.8099146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:02.8100143Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:02.8100884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:02.8101264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:02.8101780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:02.8102294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:02.8130159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:02.8206841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:02.8207724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:02.8208238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:02.8208766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:03.1614776Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:03.1666342Z   test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5662
2022-09-27T16:12:03.1699946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5663
2022-09-27T16:12:03.1728540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5664
2022-09-27T16:12:03.1756821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5665
2022-09-27T16:12:04.3388523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.3389124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.3392178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.3392819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.3741033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.3741702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.3745731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.3746333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.4132289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.4132879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.4135585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.4136174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.4445015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.4445646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.4449313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.4449969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.4683660Z dist init r=0, world=4
2022-09-27T16:12:04.5184333Z dist init r=3, world=4
2022-09-27T16:12:04.5435455Z dist init r=1, world=4
2022-09-27T16:12:04.5740290Z dist init r=2, world=4
2022-09-27T16:12:04.5949696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:04.6047664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:04.6048121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:04.6048888Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:04.6049432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:04.6050168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:04.6050758Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:04.6051907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:04.6156080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:04.6156773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:04.6157257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:04.6157784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:04.8798494Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:04.8848682Z   test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5718
2022-09-27T16:12:04.8880386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5719
2022-09-27T16:12:04.8908122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5720
2022-09-27T16:12:04.8936536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5721
2022-09-27T16:12:06.0361995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:06.0362653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:06.0365552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:06.0366095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:06.0428840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:06.0429379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:06.0433930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:06.0434546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:06.0795628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:06.0796270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:06.0799184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:06.0799815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:06.1084751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:06.1085395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:06.1088585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:06.1089223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:06.1679019Z dist init r=2, world=4
2022-09-27T16:12:06.1693802Z dist init r=0, world=4
2022-09-27T16:12:06.2065351Z dist init r=3, world=4
2022-09-27T16:12:06.2332060Z dist init r=1, world=4
2022-09-27T16:12:06.2709040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:06.2810052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:06.2812541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:06.2813415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:06.2814058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:06.2814646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:06.2815168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:06.2815685Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:06.2917813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:06.2918478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:06.2918870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:06.2919295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:06.5978024Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:06.6028781Z   test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5774
2022-09-27T16:12:06.6061524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5775
2022-09-27T16:12:06.6090101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5776
2022-09-27T16:12:06.6125555Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5777
2022-09-27T16:12:07.7935235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.7935853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.7938432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.7939001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:07.7956339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.7956946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.7960370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.7960996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:07.8251364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.8251976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.8254789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.8255444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:07.8385429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.8386029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.8390336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.8390972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:07.9210697Z dist init r=2, world=4
2022-09-27T16:12:07.9220923Z dist init r=3, world=4
2022-09-27T16:12:07.9513119Z dist init r=1, world=4
2022-09-27T16:12:07.9677271Z dist init r=0, world=4
2022-09-27T16:12:07.9921703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:08.0023050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:08.0023749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:08.0024692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:08.0025712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:08.0065319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:08.0065992Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:08.0066519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:08.0132548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:08.0133210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:08.0133567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:08.0133909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:08.3166622Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:12:08.3218921Z   test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5830
2022-09-27T16:12:08.3253440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5831
2022-09-27T16:12:08.3281497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5832
2022-09-27T16:12:08.3310190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5833
2022-09-27T16:12:09.4273446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.4274095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.4276760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.4277387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.4646905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.4647433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.4650077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.4650603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.4811554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.4812111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.4815598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.4816187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.4989743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.4990299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.4994844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.4995511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.5550063Z dist init r=3, world=4
2022-09-27T16:12:09.5913540Z dist init r=0, world=4
2022-09-27T16:12:09.6109500Z dist init r=1, world=4
2022-09-27T16:12:09.6246263Z dist init r=2, world=4
2022-09-27T16:12:09.6362037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:09.6572611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:09.6667242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:09.6667925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:09.6668677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:09.6669213Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:09.6669754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:09.6674773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:09.6680442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:09.6680940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:09.6681370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:09.6681994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:10.0351774Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:10.0403542Z   test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5886
2022-09-27T16:12:10.0438005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5887
2022-09-27T16:12:10.0466568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5888
2022-09-27T16:12:10.0495114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5889
2022-09-27T16:12:11.2092855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.2093365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.2097072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.2097664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:11.2254040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.2254622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.2258140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.2258810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:11.2528592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.2529168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.2531716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.2532342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:11.3081627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.3082293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.3085720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.3086381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:11.3374274Z dist init r=0, world=4
2022-09-27T16:12:11.3535206Z dist init r=1, world=4
2022-09-27T16:12:11.3803884Z dist init r=2, world=4
2022-09-27T16:12:11.4360656Z dist init r=3, world=4
2022-09-27T16:12:11.4468478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:11.4550739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:11.4652049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:11.4652525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:11.4653149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:11.4653701Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:11.4654242Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:11.4671687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:11.4758867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:11.4759959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:11.4760611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:11.4761146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:11.7535533Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:11.7580077Z   test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5942
2022-09-27T16:12:11.7608698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5943
2022-09-27T16:12:11.7636772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5944
2022-09-27T16:12:11.7672311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5945
2022-09-27T16:12:12.9147084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:12.9158214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:12.9159145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:12.9159780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:12.9197844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:12.9198372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:12.9202770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:12.9203417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:12.9584739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:12.9585251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:12.9588313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:12.9588938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:12.9932256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:12.9932874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:12.9935579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:12.9936204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.0427199Z dist init r=3, world=4
2022-09-27T16:12:13.0512283Z dist init r=1, world=4
2022-09-27T16:12:13.0850276Z dist init r=2, world=4
2022-09-27T16:12:13.1188007Z dist init r=0, world=4
2022-09-27T16:12:13.1462030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:13.1562751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:13.1665021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:13.1666415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:13.1667200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:13.1668104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:13.1668883Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:13.1669408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:13.1772091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:13.1772636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:13.1773197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:13.1773615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:13.4714250Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:13.4765233Z   test_nested_always_wrap_model_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5998
2022-09-27T16:12:13.4796909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5999
2022-09-27T16:12:13.4824148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6000
2022-09-27T16:12:13.4852150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6001
2022-09-27T16:12:14.6124815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:14.6125375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:14.6127913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:14.6128447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:14.6265016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:14.6265688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:14.6269249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:14.6269901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:14.6281896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:14.6282504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:14.6287272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:14.6287917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:14.6556827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:14.6557476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:14.6561176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:14.6561812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:14.7397102Z dist init r=2, world=4
2022-09-27T16:12:14.7529927Z dist init r=3, world=4
2022-09-27T16:12:14.7546955Z dist init r=0, world=4
2022-09-27T16:12:14.7847787Z dist init r=1, world=4
2022-09-27T16:12:14.8040502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:14.8140948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:14.8158854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:14.8159596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:14.8160399Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:14.8160921Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:14.8244352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:14.8244983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:14.8350435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:14.8351132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:14.8351596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:14.8352094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:15.1894210Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:15.1945540Z   test_nested_always_wrap_model_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6054
2022-09-27T16:12:15.1978861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6055
2022-09-27T16:12:15.2007672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6056
2022-09-27T16:12:15.2036275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6057
2022-09-27T16:12:16.3568295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:16.3569032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:16.3569817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:16.3570364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:16.3572023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:16.3572646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:16.3573442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:16.3574183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:16.3642732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:16.3643533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:16.3647095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:16.3647782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:16.3939113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:16.3939782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:16.3943140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:16.3943810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:16.4839771Z dist init r=3, world=4
2022-09-27T16:12:16.4840167Z dist init r=0, world=4
2022-09-27T16:12:16.4894190Z dist init r=1, world=4
2022-09-27T16:12:16.5227370Z dist init r=2, world=4
2022-09-27T16:12:16.5348667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:16.5538227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:16.5639073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:16.5639472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:16.5640127Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:16.5640664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:16.5641202Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:16.5653019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:16.5746743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:16.5747309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:16.5747873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:16.5748340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:16.9077514Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:16.9129699Z   test_nested_always_wrap_model_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6110
2022-09-27T16:12:16.9162997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6111
2022-09-27T16:12:16.9191233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6112
2022-09-27T16:12:16.9223359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6113
2022-09-27T16:12:18.0974235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.0974801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.0977762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.0978280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:18.1347362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.1348170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.1350678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.1351412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:18.1377340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.1377920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.1382244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.1382868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:18.1745092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.1745715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.1748584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.1749219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:18.2252550Z dist init r=0, world=4
2022-09-27T16:12:18.2625269Z dist init r=2, world=4
2022-09-27T16:12:18.2649655Z dist init r=3, world=4
2022-09-27T16:12:18.3039491Z dist init r=1, world=4
2022-09-27T16:12:18.3350861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:18.3451491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:18.3553591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:18.3554332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:18.3555188Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:18.3555785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:18.3556381Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:18.3556975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:18.3660592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:18.3661286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:18.3661767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:18.3662120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:18.6263984Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:18.6314547Z   test_nested_always_wrap_model_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6166
2022-09-27T16:12:18.6347665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6167
2022-09-27T16:12:18.6375073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6168
2022-09-27T16:12:18.6402860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6169
2022-09-27T16:12:19.8059702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.8060241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.8063126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.8064019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.8181542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.8182347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.8185626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.8186245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.8380475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.8381066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.8385597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.8386239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.8432698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.8433295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.8436871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.8437497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.9325166Z dist init r=2, world=4
2022-09-27T16:12:19.9479580Z dist init r=0, world=4
2022-09-27T16:12:19.9639099Z dist init r=3, world=4
2022-09-27T16:12:19.9681348Z dist init r=1, world=4
2022-09-27T16:12:19.9992076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:20.0093553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:20.0195601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:20.0196469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:20.0197348Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:20.0198132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:20.0199006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:20.0199535Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:20.0203324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:20.0204034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:20.0204494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:20.0204931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:20.3444496Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:20.3497749Z   test_nested_always_wrap_model_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6222
2022-09-27T16:12:20.3532305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6223
2022-09-27T16:12:20.3560465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6224
2022-09-27T16:12:20.3589096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6225
2022-09-27T16:12:21.5002942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:21.5003489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:21.5004834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:21.5005505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:21.5220989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:21.5221578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:21.5224853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:21.5225487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:21.5271746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:21.5272374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:21.5275984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:21.5276653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:21.5418919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:21.5419495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:21.5423675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:21.5424509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:21.6269022Z dist init r=2, world=4
2022-09-27T16:12:21.6489360Z dist init r=0, world=4
2022-09-27T16:12:21.6534417Z dist init r=1, world=4
2022-09-27T16:12:21.6699459Z dist init r=3, world=4
2022-09-27T16:12:21.6844944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:21.6946502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:21.7048571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:21.7049157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:21.7050093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:21.7050851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:21.7051615Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:21.7052454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:21.7055607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:21.7056364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:21.7056790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:21.7057235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:21.9629139Z skip: Need at least 2 CUDA devices (1.618s)
2022-09-27T16:12:21.9680163Z   test_nested_always_wrap_model_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6278
2022-09-27T16:12:21.9713342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6279
2022-09-27T16:12:21.9741444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6280
2022-09-27T16:12:21.9769669Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6281
2022-09-27T16:12:23.1479690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:23.1480551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:23.1483489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:23.1484072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:23.1646608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:23.1647290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:23.1651516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:23.1652162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:23.1757047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:23.1757659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:23.1761492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:23.1762120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:23.1822520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:23.1823105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:23.1828083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:23.1828725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:23.2778235Z dist init r=0, world=4
2022-09-27T16:12:23.2946955Z dist init r=2, world=4
2022-09-27T16:12:23.3059683Z dist init r=3, world=4
2022-09-27T16:12:23.3175109Z dist init r=1, world=4
2022-09-27T16:12:23.3457460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:23.3490626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:23.3491644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:23.3492866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:23.3493533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:23.3494071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:23.3494595Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:23.3559500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:23.3598443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:23.3599176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:23.3599731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:23.3600288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:23.6811642Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:23.6863738Z   test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6334
2022-09-27T16:12:23.6898784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6335
2022-09-27T16:12:23.6927819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6336
2022-09-27T16:12:23.6956589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6337
2022-09-27T16:12:24.8562908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:24.8563830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:24.8564740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:24.8565403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:24.8644916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:24.8645475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:24.8650126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:24.8650829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:24.9078271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:24.9078920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:24.9081152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:24.9081791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:24.9142807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:24.9143433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:24.9148095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:24.9148763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:24.9811712Z dist init r=1, world=4
2022-09-27T16:12:24.9916126Z dist init r=0, world=4
2022-09-27T16:12:25.0372178Z dist init r=3, world=4
2022-09-27T16:12:25.0395458Z dist init r=2, world=4
2022-09-27T16:12:25.0605103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:25.0706776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:25.0807868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:25.0808454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:25.0809375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:25.0810191Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:25.0811012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:25.0811793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:25.0816575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:25.0817127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:25.0817681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:25.0818389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:25.3997198Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:25.4047025Z   test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6390
2022-09-27T16:12:25.4079819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6391
2022-09-27T16:12:25.4107269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6392
2022-09-27T16:12:25.4134529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6393
2022-09-27T16:12:26.5330631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:26.5331258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:26.5334331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:26.5334987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:26.5691580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:26.5692215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:26.5695124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:26.5695743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:26.6031507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:26.6032155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:26.6035138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:26.6035825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:26.6454252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:26.6457551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:26.6458415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:26.6459023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:26.6609391Z dist init r=3, world=4
2022-09-27T16:12:26.6997140Z dist init r=0, world=4
2022-09-27T16:12:26.7292475Z dist init r=2, world=4
2022-09-27T16:12:26.7712429Z dist init r=1, world=4
2022-09-27T16:12:26.7925958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:26.7926434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:26.8027575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:26.8028101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:26.8029015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:26.8029675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:26.8030202Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:26.8030720Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:26.8134520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:26.8135107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:26.8135649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:26.8136291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:27.1176486Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:27.1226623Z   test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6446
2022-09-27T16:12:27.1260423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6447
2022-09-27T16:12:27.1288054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6448
2022-09-27T16:12:27.1316903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6449
2022-09-27T16:12:28.3109496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:28.3110183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:28.3112980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:28.3113672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:28.3230711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:28.3231273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:28.3235688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:28.3236328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:28.3539400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:28.3540052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:28.3542854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:28.3543493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:28.3639828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:28.3640421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:28.3644103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:28.3644787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:28.4503818Z dist init r=1, world=4
2022-09-27T16:12:28.4615598Z dist init r=2, world=4
2022-09-27T16:12:28.4802620Z dist init r=3, world=4
2022-09-27T16:12:28.4919452Z dist init r=0, world=4
2022-09-27T16:12:28.5125761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:28.5315355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:28.5315847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:28.5316307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:28.5316934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:28.5317469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:28.5317979Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:28.5328358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:28.5422793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:28.5423554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:28.5424379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:28.5424916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:28.8358276Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:28.8408495Z   test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6502
2022-09-27T16:12:28.8441439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6503
2022-09-27T16:12:28.8470080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6504
2022-09-27T16:12:28.8498135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6505
2022-09-27T16:12:29.9804014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:29.9804722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:29.9807666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:29.9808338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.0107988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.0108695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.0110890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.0111488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.0315700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.0316396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.0320450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.0321172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.0977302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.0978083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.0980587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.0981210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.1079917Z dist init r=0, world=4
2022-09-27T16:12:30.1375753Z dist init r=1, world=4
2022-09-27T16:12:30.1572715Z dist init r=3, world=4
2022-09-27T16:12:30.2257094Z dist init r=2, world=4
2022-09-27T16:12:30.2466973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:30.2497152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:30.2598383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:30.2599082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:30.2599759Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:30.2600452Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:30.2600989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:30.2670646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:30.2706841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:30.2707390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:30.2708469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:30.2708967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:30.5538332Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:30.5582519Z   test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6558
2022-09-27T16:12:30.5612977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6559
2022-09-27T16:12:30.5641204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6560
2022-09-27T16:12:30.5676910Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6561
2022-09-27T16:12:31.7110131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:31.7110750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:31.7114174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:31.7114881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:31.7767746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:31.7768262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:31.7772172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:31.7772838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:31.8114020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:31.8114656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:31.8117319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:31.8117970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:31.8253373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:31.8254029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:31.8257265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:31.8257918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:31.8389191Z dist init r=1, world=4
2022-09-27T16:12:31.9034687Z dist init r=3, world=4
2022-09-27T16:12:31.9368065Z dist init r=0, world=4
2022-09-27T16:12:31.9541531Z dist init r=2, world=4
2022-09-27T16:12:31.9744866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:31.9807005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:31.9807471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:31.9808111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:31.9809099Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:31.9809652Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:31.9810243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:31.9847339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:31.9916725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:31.9917270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:31.9917813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:31.9918337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:32.2718179Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:32.2769247Z   test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6614
2022-09-27T16:12:32.2801265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6615
2022-09-27T16:12:32.2830459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6616
2022-09-27T16:12:32.2858180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6617
2022-09-27T16:12:33.4124173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.4124702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.4128083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.4128641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:33.4452152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.4452821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.4455206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.4455844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:33.4564813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.4565471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.4568301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.4568907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:33.4653767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.4654376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.4658159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.4658774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:33.5400556Z dist init r=2, world=4
2022-09-27T16:12:33.5740319Z dist init r=0, world=4
2022-09-27T16:12:33.5825494Z dist init r=1, world=4
2022-09-27T16:12:33.5951794Z dist init r=3, world=4
2022-09-27T16:12:33.6151913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:33.6152320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:33.6253385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:33.6254083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:33.6255197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:33.6256028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:33.6256816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:33.6257627Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:33.6260731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:33.6261375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:33.6261790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:33.6262144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:33.9899563Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:33.9951827Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6670
2022-09-27T16:12:33.9985793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6671
2022-09-27T16:12:34.0014079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6672
2022-09-27T16:12:34.0042458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6673
2022-09-27T16:12:35.0894708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.0895369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.0899324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.0899944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.1522679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.1523293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.1525919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.1526565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.1530893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.1531508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.1535555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.1536217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.1754650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.1755284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.1756705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.1757356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.2168958Z dist init r=2, world=4
2022-09-27T16:12:35.2792060Z dist init r=1, world=4
2022-09-27T16:12:35.2794771Z dist init r=0, world=4
2022-09-27T16:12:35.3049142Z dist init r=3, world=4
2022-09-27T16:12:35.3182059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:35.3283351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:35.3385216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:35.3386213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:35.3386881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:35.3388728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:35.3389889Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:35.3390964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:35.3393011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:35.3393471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:35.3393943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:35.3394291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:35.6080173Z skip: Need at least 2 CUDA devices (1.618s)
2022-09-27T16:12:35.6133204Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6726
2022-09-27T16:12:35.6166296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6727
2022-09-27T16:12:35.6194235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6728
2022-09-27T16:12:35.6223290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6729
2022-09-27T16:12:36.7845560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:36.7846228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:36.7849748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:36.7850390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:36.8304542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:36.8305182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:36.8308168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:36.8308794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:36.8333709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:36.8334276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:36.8338296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:36.8338927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:36.8415095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:36.8415691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:36.8419780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:36.8420422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:36.9147448Z dist init r=2, world=4
2022-09-27T16:12:36.9564984Z dist init r=0, world=4
2022-09-27T16:12:36.9597014Z dist init r=1, world=4
2022-09-27T16:12:36.9710643Z dist init r=3, world=4
2022-09-27T16:12:36.9819503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:36.9908230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:36.9909114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:36.9909663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:36.9910041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:36.9910555Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:36.9911081Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:36.9921246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:37.0016378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:37.0016994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:37.0017510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:37.0017969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:37.3263934Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:37.3315789Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6782
2022-09-27T16:12:37.3350301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6783
2022-09-27T16:12:37.3379143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6784
2022-09-27T16:12:37.3407286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6785
2022-09-27T16:12:38.4905131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:38.4905761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:38.4908384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:38.4908997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:38.4952660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:38.4953213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:38.4957259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:38.4957862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:38.5198104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:38.5198715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:38.5202060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:38.5202673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:38.5228533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:38.5229155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:38.5233160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:38.5233781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:38.6170684Z dist init r=1, world=4
2022-09-27T16:12:38.6216235Z dist init r=2, world=4
2022-09-27T16:12:38.6479313Z dist init r=0, world=4
2022-09-27T16:12:38.6498175Z dist init r=3, world=4
2022-09-27T16:12:38.6781118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:38.6882680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:38.6883764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:38.6884503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:38.6885423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:38.6886160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:38.6887053Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:38.6888015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:38.6892253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:38.6892789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:38.6893343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:38.6893862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:39.0449401Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:39.0501817Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6838
2022-09-27T16:12:39.0534902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6839
2022-09-27T16:12:39.0562705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6840
2022-09-27T16:12:39.0590908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6841
2022-09-27T16:12:40.2143430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:40.2144382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:40.2147157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:40.2147832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:40.2201865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:40.2202467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:40.2206529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:40.2207187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:40.2342543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:40.2343134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:40.2346952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:40.2347566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:40.2861007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:40.2861649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:40.2865275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:40.2865902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:40.3418321Z dist init r=3, world=4
2022-09-27T16:12:40.3476099Z dist init r=0, world=4
2022-09-27T16:12:40.3604429Z dist init r=1, world=4
2022-09-27T16:12:40.4148389Z dist init r=2, world=4
2022-09-27T16:12:40.4389879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:40.4390284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:40.4492008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:40.4492662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:40.4493530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:40.4494172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:40.4494703Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:40.4495209Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:40.4597699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:40.4598606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:40.4599498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:40.4600331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:40.7632629Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:40.7685549Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6894
2022-09-27T16:12:40.7718974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6895
2022-09-27T16:12:40.7748353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6896
2022-09-27T16:12:40.7777063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6897
2022-09-27T16:12:41.9284026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:41.9284705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:41.9288303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:41.9288943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:41.9481580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:41.9482257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:41.9485637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:41.9486544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:41.9940483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:41.9941342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:41.9944149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:41.9944885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:42.0019238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:42.0019852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:42.0023066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:42.0023723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:42.0739924Z dist init r=1, world=4
2022-09-27T16:12:42.0767347Z dist init r=2, world=4
2022-09-27T16:12:42.1260019Z dist init r=0, world=4
2022-09-27T16:12:42.1308061Z dist init r=3, world=4
2022-09-27T16:12:42.1551934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:42.1653922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:42.1654900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:42.1655524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:42.1656051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:42.1656693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:42.1657227Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:42.1661102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:42.1666825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:42.1667531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:42.1668092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:42.1668625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:42.4818832Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:42.4871022Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6950
2022-09-27T16:12:42.4903801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6951
2022-09-27T16:12:42.4932698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6952
2022-09-27T16:12:42.4960940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6953
2022-09-27T16:12:43.6774975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:43.6775498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:43.6778981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:43.6779676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:43.6780506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:43.6781345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:43.6784195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:43.6785032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:43.7241059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:43.7241652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:43.7244294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:43.7244881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:43.7393045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:43.7393677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:43.7396793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:43.7397454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:43.8044091Z dist init r=1, world=4
2022-09-27T16:12:43.8050393Z dist init r=3, world=4
2022-09-27T16:12:43.8509748Z dist init r=2, world=4
2022-09-27T16:12:43.8683352Z dist init r=0, world=4
2022-09-27T16:12:43.8863077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:43.9065799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:43.9168013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:43.9168599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:43.9169561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:43.9170183Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:43.9170754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:43.9171283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:43.9175544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:43.9176211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:43.9178914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:43.9179291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:44.2002747Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:44.2054557Z   test_transformer_offload_false_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7006
2022-09-27T16:12:44.2088142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7007
2022-09-27T16:12:44.2116498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7008
2022-09-27T16:12:44.2152292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7009
2022-09-27T16:12:45.3641621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:45.3642278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:45.3644976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:45.3645816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:45.3669161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:45.3670041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:45.3674756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:45.3675386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:45.4044154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:45.4044763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:45.4046938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:45.4047578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:45.4201967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:45.4202591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:45.4206489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:45.4207151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:45.4911585Z dist init r=2, world=4
2022-09-27T16:12:45.4979576Z dist init r=3, world=4
2022-09-27T16:12:45.5310656Z dist init r=1, world=4
2022-09-27T16:12:45.5451686Z dist init r=0, world=4
2022-09-27T16:12:45.5822153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:45.5822768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:45.5823572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:45.5824207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:45.5824830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:45.5826797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:45.5827335Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:45.5827857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:45.5930920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:45.5931658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:45.5932185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:45.5932737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:45.9193463Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:12:45.9245167Z   test_transformer_offload_false_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7062
2022-09-27T16:12:45.9279381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7063
2022-09-27T16:12:45.9308104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7064
2022-09-27T16:12:45.9336353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7065
2022-09-27T16:12:47.1024732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.1028229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.1028947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.1029586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.1030249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.1030729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.1035149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.1035696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.1229868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.1230494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.1232900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.1233514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.1354665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.1355277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.1359831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.1360483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.2295043Z dist init r=1, world=4
2022-09-27T16:12:47.2301757Z dist init r=0, world=4
2022-09-27T16:12:47.2635040Z dist init r=3, world=4
2022-09-27T16:12:47.2670976Z dist init r=2, world=4
2022-09-27T16:12:47.2880360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:47.2906845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:47.3008098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:47.3008724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:47.3009502Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:47.3010050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:47.3010578Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:47.3083430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:47.3117221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:47.3117840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:47.3136993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:47.3137423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:47.6377303Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:47.6429731Z   test_transformer_offload_false_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7118
2022-09-27T16:12:47.6463746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7119
2022-09-27T16:12:47.6493157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7120
2022-09-27T16:12:47.6521485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7121
2022-09-27T16:12:48.8235498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:48.8236395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:48.8238538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:48.8239157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:48.8430462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:48.8431030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:48.8434375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:48.8434957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:48.8924506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:48.8925111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:48.8927971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:48.8928600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:48.9293211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:48.9293880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:48.9297322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:48.9298020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:48.9520636Z dist init r=1, world=4
2022-09-27T16:12:48.9705679Z dist init r=0, world=4
2022-09-27T16:12:49.0180362Z dist init r=2, world=4
2022-09-27T16:12:49.0575841Z dist init r=3, world=4
2022-09-27T16:12:49.0736704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:49.0737207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:49.0738010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:49.0738865Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:49.0739391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:49.0739908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:49.0740483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:49.0838997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:49.0846472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:49.0847053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:49.0847610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:49.0848160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:49.3562810Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:49.3614628Z   test_transformer_offload_true_no_shard_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7174
2022-09-27T16:12:49.3647762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7175
2022-09-27T16:12:49.3677431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7176
2022-09-27T16:12:49.3714183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7177
2022-09-27T16:12:50.5281213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:50.5281784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:50.5284524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:50.5285113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:50.5285812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:50.5286379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:50.5289305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:50.5289863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:50.5459360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:50.5459904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:50.5464386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:50.5465007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:50.5473529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:50.5474260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:50.5477794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:50.5478414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:50.6558550Z dist init r=0, world=4
2022-09-27T16:12:50.6561116Z dist init r=3, world=4
2022-09-27T16:12:50.6718388Z dist init r=2, world=4
2022-09-27T16:12:50.6762562Z dist init r=1, world=4
2022-09-27T16:12:50.7027965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:50.7129053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:50.7231336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:50.7232081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:50.7233030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:50.7233654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:50.7234284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:50.7234796Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:50.7242178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:50.7242880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:50.7243261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:50.7243779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:51.0756167Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:12:51.0810254Z   test_transformer_offload_true_none_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7230
2022-09-27T16:12:51.0844534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7231
2022-09-27T16:12:51.0873523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7232
2022-09-27T16:12:51.0902097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7233
2022-09-27T16:12:52.2616539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:52.2617168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:52.2620146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:52.2620768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:52.2656872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:52.2657525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:52.2661492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:52.2662107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:52.2907743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:52.2908375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:52.2911548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:52.2912139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:52.3265529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:52.3266194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:52.3269600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:52.3270221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:52.3894147Z dist init r=2, world=4
2022-09-27T16:12:52.3924901Z dist init r=1, world=4
2022-09-27T16:12:52.4168405Z dist init r=3, world=4
2022-09-27T16:12:52.4666866Z dist init r=0, world=4
2022-09-27T16:12:52.4938876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:52.5040231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:52.5141281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:52.5143793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:52.5144739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:52.5145437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:52.5145975Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:52.5146504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:52.5149742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:52.5150470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:52.5150842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:52.5151661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:52.7942797Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:12:52.7997959Z   test_transformer_offload_true_shard_grad_op_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7286
2022-09-27T16:12:52.8030489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7287
2022-09-27T16:12:52.8058277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7288
2022-09-27T16:12:52.8086848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7289
2022-09-27T16:12:53.9775843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:53.9776462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:53.9779690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:53.9780434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:53.9872241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:53.9872863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:53.9873617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:53.9874157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:53.9875752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:53.9876418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:53.9877223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:53.9877793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:54.0033882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:54.0034457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:54.0038967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:54.0039577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:54.1045462Z dist init r=2, world=4
2022-09-27T16:12:54.1137944Z dist init r=0, world=4
2022-09-27T16:12:54.1138297Z dist init r=3, world=4
2022-09-27T16:12:54.1314697Z dist init r=1, world=4
2022-09-27T16:12:54.1556104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:54.1656418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:54.1758810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:12:54.1759647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:12:54.1760690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:54.1761424Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:54.1762131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:54.1763181Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:12:54.1766633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:12:54.1767320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:54.1767829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:54.1768403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:12:54.5129217Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:12:54.5129480Z 
2022-09-27T16:12:54.5129818Z ----------------------------------------------------------------------
2022-09-27T16:12:54.5130070Z Ran 59 tests in 93.028s
2022-09-27T16:12:54.5130221Z 
2022-09-27T16:12:54.5130328Z OK (skipped=59)
2022-09-27T16:12:54.5130437Z 
2022-09-27T16:12:54.5130511Z Generating XML reports...
2022-09-27T16:12:54.5171445Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220927161121.xml
2022-09-27T16:12:54.5175434Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220927161121.xml
2022-09-27T16:12:54.5180113Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220927161121.xml
2022-09-27T16:12:54.5238221Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220927161121.xml
2022-09-27T16:12:54.8145447Z Running distributed/test_c10d_pypg ... [2022-09-27 16:12:54.814104]
2022-09-27T16:12:54.8146019Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:12:54.814178]
2022-09-27T16:12:56.0468492Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg
2022-09-27T16:12:56.0484238Z 
2022-09-27T16:12:56.0484881Z Running tests...
2022-09-27T16:12:56.0485561Z ----------------------------------------------------------------------
2022-09-27T16:12:56.0494613Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:12:56.2886216Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7353
2022-09-27T16:12:57.2891914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:57.2892600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:57.3183305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:57.3184168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:57.4507783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:57.6915431Z skip: Need at least 2 CUDA devices (1.643s)
2022-09-27T16:12:57.6921886Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:12:57.6966764Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7364
2022-09-27T16:12:58.6934664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:58.6935055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:58.7248357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:58.7248728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:58.8579379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:59.0993975Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:12:59.1002456Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:12:59.1046038Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7375
2022-09-27T16:13:00.1293168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:00.1293543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:00.1573916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:00.1574272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:00.2900104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:00.5072511Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:00.5081075Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:00.5123869Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7386
2022-09-27T16:13:01.5664885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:01.5665290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:01.5952439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:01.5952831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:01.7327313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:02.0153478Z skip: Need at least 2 CUDA devices (1.508s)
2022-09-27T16:13:02.0159197Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:02.0203486Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7397
2022-09-27T16:13:03.0208892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:03.0209556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:03.0518020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:03.0518644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:03.1844458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:03.4231085Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:03.4237554Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:03.4280565Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7408
2022-09-27T16:13:04.4391559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:04.4391941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:04.4695474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:04.4695846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:04.6023646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:04.8308681Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:04.8317944Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:04.8359894Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7419
2022-09-27T16:13:05.8349189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:05.8349781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:05.8646676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:05.8647038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:05.9969338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:06.2387930Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:06.2397229Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:06.2440594Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7430
2022-09-27T16:13:07.2476270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:07.2476971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:07.2777975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:07.2778591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:07.4113040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:07.6468372Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:07.6474672Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:07.6519377Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7441
2022-09-27T16:13:08.6539987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:08.6540382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:08.6843557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:08.6843988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:08.8179952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:09.0546197Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:09.0556308Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:09.0599018Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7452
2022-09-27T16:13:10.0631230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:10.0631612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:10.0915472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:10.0915855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:10.2238327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:10.4626938Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:10.4636764Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:10.4679249Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7463
2022-09-27T16:13:11.4730998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:11.4731372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:11.5021187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:11.5021555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:11.6350336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:11.8711411Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:11.8721573Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:11.8764726Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7474
2022-09-27T16:13:12.8813056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:12.8813458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:12.9112212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:12.9112601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:13.0456967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:13.2795133Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:13.2805782Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-09-27T16:13:13.2848694Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7485
2022-09-27T16:13:14.3076076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:14.3076460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:14.3365321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:14.4719094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:14.4719465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:14.6875951Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:14.6929027Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7496
2022-09-27T16:13:15.7149772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:15.7150218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:15.7434589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:15.7434963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:15.8756770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:15.8824538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw70gs3o
2022-09-27T16:13:15.8826405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw70gs3o/_remote_module_non_scriptable.py
2022-09-27T16:13:16.0956601Z ok (1.408s)
2022-09-27T16:13:16.1005291Z   test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7507
2022-09-27T16:13:17.1009037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:17.1009441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:17.1300055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:17.1300459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:17.2628591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:17.2697032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr4q4qzn
2022-09-27T16:13:17.2699123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr4q4qzn/_remote_module_non_scriptable.py
2022-09-27T16:13:17.2836616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:17.5033278Z ok (1.408s)
2022-09-27T16:13:17.5081904Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7518
2022-09-27T16:13:18.5074666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:18.5075070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:18.5388569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:18.5388945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:18.6708967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:18.6777266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88uho904
2022-09-27T16:13:18.6778808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88uho904/_remote_module_non_scriptable.py
2022-09-27T16:13:18.6916558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:18.9109863Z ok (1.407s)
2022-09-27T16:13:18.9160285Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7529
2022-09-27T16:13:19.9195948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:19.9196374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:19.9479367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:19.9479739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:20.0795410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:20.0799765Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.0800965Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.0802040Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.0803202Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.0804183Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.0805201Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:20.3188045Z ok (1.408s)
2022-09-27T16:13:20.3241797Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7540
2022-09-27T16:13:21.3276134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:21.3276526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:21.3575487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:21.3575885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:21.4895343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:21.7270336Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:21.7326892Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7551
2022-09-27T16:13:22.7343369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:22.7343750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:22.7632460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:22.7632849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:22.8981809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:23.1353985Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:23.1361233Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:23.1405427Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7562
2022-09-27T16:13:24.1376289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:24.1377034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:24.1668415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:24.1669059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:24.3008221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:24.5433520Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:24.5440239Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:24.5485006Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7573
2022-09-27T16:13:25.5454041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:25.5454418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:25.5745151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:25.5745548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:25.7055299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:25.9513072Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:25.9521767Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:25.9565991Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7584
2022-09-27T16:13:26.9549494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:26.9549865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:26.9847119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:26.9847486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:27.1168364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:27.3594135Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:27.3602268Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:27.3643527Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7595
2022-09-27T16:13:28.3890692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:28.3891414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:28.4181705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:28.4182340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:28.5515945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:28.7670804Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:28.7677795Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:28.7720493Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7606
2022-09-27T16:13:29.7732232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:29.7732911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:29.8015531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:29.8016201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:29.9338469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:30.1748087Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:30.1755013Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:30.1799297Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7617
2022-09-27T16:13:31.1888947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:31.1889358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:31.2200564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:31.3524758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:31.3525285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:31.5827204Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:31.5837289Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:31.5881896Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7628
2022-09-27T16:13:32.5861823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:32.5862233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:32.6162870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:32.6163292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:32.7487148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:32.9910128Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:32.9919453Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:32.9964050Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7639
2022-09-27T16:13:34.0016957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:34.0017335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:34.0315186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:34.0315561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:34.1652413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:34.3991760Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:34.3998372Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:34.4042586Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7650
2022-09-27T16:13:35.4031967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:35.4032330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:35.4312941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:35.4313373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:35.5643162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:35.8070690Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:35.8080595Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:35.8124609Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7661
2022-09-27T16:13:36.8116565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:36.8116979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:36.8403632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:36.8404065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:36.9734064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:37.2153362Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:37.2162295Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:37.2205755Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7672
2022-09-27T16:13:38.2172784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:38.2173447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:38.2488973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:38.2489584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:38.3822242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:38.6234430Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:38.6245424Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:38.6287604Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7683
2022-09-27T16:13:39.6513123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:39.6513499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:39.6804635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:39.6805001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:39.8129410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:40.0315502Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:40.0326765Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-09-27T16:13:40.0373400Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7694
2022-09-27T16:13:41.0339519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:41.0339988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:41.0647236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:41.0647592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:41.1976844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:41.4401286Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:41.4455493Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7705
2022-09-27T16:13:42.4458284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:42.4458670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:42.4744533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:42.4744921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:42.6077298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:42.6149647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgt3bdh5b
2022-09-27T16:13:42.6151328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgt3bdh5b/_remote_module_non_scriptable.py
2022-09-27T16:13:42.8483269Z ok (1.408s)
2022-09-27T16:13:42.8532684Z   test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7716
2022-09-27T16:13:43.8520658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:43.8521245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:43.8810763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:43.8811144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:44.0152526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:44.0220964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpomdt7d
2022-09-27T16:13:44.0222371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpomdt7d/_remote_module_non_scriptable.py
2022-09-27T16:13:44.0360730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:44.2560365Z ok (1.408s)
2022-09-27T16:13:44.2609415Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7727
2022-09-27T16:13:45.2616092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:45.2616794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:45.2914360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:45.2914971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:45.4244995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:45.4314189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyett9tl
2022-09-27T16:13:45.4315738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyett9tl/_remote_module_non_scriptable.py
2022-09-27T16:13:45.4452656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:45.6637109Z ok (1.407s)
2022-09-27T16:13:45.6687682Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7738
2022-09-27T16:13:46.6633358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:46.6633748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:46.6923942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:46.6924318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:46.8281796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:46.8285950Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:46.8287322Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:46.8288542Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:46.8289354Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:46.8290227Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:46.8304854Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:13:47.0715655Z ok (1.408s)
2022-09-27T16:13:47.0773018Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7749
2022-09-27T16:13:48.0746165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:48.0746541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:48.1034933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:48.1035488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:48.2361204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:48.4800798Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:48.4857406Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7760
2022-09-27T16:13:49.4843616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:49.4843994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:49.5134064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:49.5134440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:49.6464305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:49.8885032Z skip: Need at least 2 CUDA devices (1.408s)
2022-09-27T16:13:49.8885235Z 
2022-09-27T16:13:49.8886796Z ----------------------------------------------------------------------
2022-09-27T16:13:49.8887189Z Ran 38 tests in 53.840s
2022-09-27T16:13:49.8887311Z 
2022-09-27T16:13:49.8887376Z OK (skipped=30)
2022-09-27T16:13:49.8887495Z 
2022-09-27T16:13:49.8887582Z Generating XML reports...
2022-09-27T16:13:49.8953309Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20220927161256.xml
2022-09-27T16:13:49.8972330Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20220927161256.xml
2022-09-27T16:13:50.1760743Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-09-27 16:13:50.175650]
2022-09-27T16:13:50.1761359Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:13:50.175727]
2022-09-27T16:13:51.5518763Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-09-27T16:13:51.5533375Z 
2022-09-27T16:13:51.5533478Z Running tests...
2022-09-27T16:13:51.5534652Z ----------------------------------------------------------------------
2022-09-27T16:13:51.5550841Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:51.7795850Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.226s)
2022-09-27T16:13:51.7809173Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:51.7896474Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7782
2022-09-27T16:13:51.7924218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7783
2022-09-27T16:13:53.0472967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:53.0473399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:53.0475248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:53.0475810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:53.0685017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:53.0685381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:53.0688728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:53.0689129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:53.1714315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:53.1924946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:53.4964491Z skip: CUDA is not available. (1.717s)
2022-09-27T16:13:53.4976606Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.4977529Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.4987876Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.4988678Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.4998431Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.4998980Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5008748Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5009708Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5020011Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5021203Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5032075Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5033194Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5043324Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5043927Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5053985Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5054496Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5064195Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5065155Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5074865Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5075769Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5085276Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5085790Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5095782Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5096288Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5106220Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5106757Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5116490Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5116995Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5126568Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5127076Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5137261Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5137932Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:13:53.5167315Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:53.5211454Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7804
2022-09-27T16:13:53.5245090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7805
2022-09-27T16:13:54.7607906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:54.7608286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:54.7611370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:54.7611745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:54.7770768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:54.7771298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:54.7774668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:54.7775209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:54.8832953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:54.8994764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:55.1281624Z skip: CUDA is not available. (1.614s)
2022-09-27T16:13:55.1314261Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:55.1358401Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7826
2022-09-27T16:13:55.1390865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7827
2022-09-27T16:13:56.3965759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:56.3966142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:56.3968837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:56.3969202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:56.4469809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:56.4470205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:56.4473700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:56.4474277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:56.5199649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:56.5725467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:56.8424756Z skip: CUDA is not available. (1.714s)
2022-09-27T16:13:56.8457248Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:56.8503006Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7848
2022-09-27T16:13:56.8536395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7849
2022-09-27T16:13:58.1222434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:58.1223073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:58.1224759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:58.1225336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:58.1346054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:58.1346736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:58.1350680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:58.1351338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:58.2451871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:58.2603313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:58.5571340Z skip: CUDA is not available. (1.714s)
2022-09-27T16:13:58.5619744Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:13:58.5666503Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7870
2022-09-27T16:13:58.5699478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7871
2022-09-27T16:13:59.8168896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:59.8169544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:59.8172852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:59.8173523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:59.8737350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:59.8738006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:59.8740068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:59.8740711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:59.9397173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:59.9965373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:00.2731887Z skip: CUDA is not available. (1.716s)
2022-09-27T16:14:00.2764890Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:00.2808693Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7892
2022-09-27T16:14:00.2840408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7893
2022-09-27T16:14:01.5263635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:01.5264191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:01.5266679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:01.5267239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:01.5284058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:01.5284543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:01.5288509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:01.5288895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:01.6487170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:01.6501957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:01.8874740Z skip: CUDA is not available. (1.614s)
2022-09-27T16:14:01.8906975Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:01.8954013Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7914
2022-09-27T16:14:01.8986384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7915
2022-09-27T16:14:03.1708353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:03.1708928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:03.1710954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:03.1711517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:03.1755338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:03.1755713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:03.1758998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:03.1759350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:03.2938472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:03.3014407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:03.6021941Z skip: CUDA is not available. (1.715s)
2022-09-27T16:14:03.6033259Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:03.6078693Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7936
2022-09-27T16:14:03.6111804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7937
2022-09-27T16:14:04.8613862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:04.8614526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:04.8617985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:04.8618643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:04.8720997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:04.8721642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:04.8725011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:04.8725640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:04.9844585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:04.9944364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:05.2146642Z skip: CUDA is not available. (1.612s)
2022-09-27T16:14:05.2167243Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:05.2207568Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7958
2022-09-27T16:14:05.2237863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7959
2022-09-27T16:14:06.5000429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:06.5001123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:06.5003133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:06.5003790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:06.5267045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:06.5267718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:06.5271378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:06.5271980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:06.6234230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:06.6511667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:06.9272872Z skip: CUDA is not available. (1.713s)
2022-09-27T16:14:06.9296662Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:06.9341157Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7980
2022-09-27T16:14:06.9374043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7981
2022-09-27T16:14:08.2066961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:08.2067555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:08.2069541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:08.2070131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:08.2302043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:08.2302729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:08.2306495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:08.2307137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:08.3436454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:08.3437236Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-09-27T16:14:08.3581605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:08.3582524Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-09-27T16:14:08.6409065Z ok (1.713s)
2022-09-27T16:14:08.6417430Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:08.6422307Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:14:08.6435617Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:08.6479765Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8002
2022-09-27T16:14:08.6512495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8003
2022-09-27T16:14:09.8899840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:09.8900227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:09.8902635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:09.8902999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:09.9142555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:09.9143046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:09.9146514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:09.9147003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:10.0127832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:10.0396432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:10.2543705Z skip: CUDA is not available. (1.612s)
2022-09-27T16:14:10.2561562Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:10.2605288Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8024
2022-09-27T16:14:10.2637283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8025
2022-09-27T16:14:11.5006970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:11.5007336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:11.5009887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:11.5010260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:11.5136198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:11.5136803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:11.5140546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:11.5141058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:11.6233908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:11.6359346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:11.8671563Z skip: CUDA is not available. (1.613s)
2022-09-27T16:14:11.8675385Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:11.8718526Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8046
2022-09-27T16:14:11.8750422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8047
2022-09-27T16:14:13.1062343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:13.1062714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:13.1065499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:13.1065864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:13.1339417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:13.1339809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:13.1343423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:13.1343794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:13.2289494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:13.2588554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:13.2703124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:13.2703795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:13.2704618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:13.2705140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:13.2770020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas5meri_
2022-09-27T16:14:13.2771736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph58qdcre
2022-09-27T16:14:13.2772505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas5meri_/_remote_module_non_scriptable.py
2022-09-27T16:14:13.2774343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph58qdcre/_remote_module_non_scriptable.py
2022-09-27T16:14:13.2931237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:13.2931928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:13.3256166Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpas5meri_'>
2022-09-27T16:14:13.3256834Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:14:13.3257704Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmph58qdcre'>
2022-09-27T16:14:13.3258339Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:14:13.5785341Z ok (1.711s)
2022-09-27T16:14:13.5789553Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:13.5790361Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:13.5795899Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:13.5839253Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8074
2022-09-27T16:14:13.5871699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8075
2022-09-27T16:14:14.8247771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:14.8248182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:14.8250492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:14.8250855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:14.8356770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:14.8357157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:14.8360699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:14.8361364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:14.9475922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:14.9573974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:15.1905634Z skip: Need at least 4 CUDA devices (1.611s)
2022-09-27T16:14:15.1912374Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:14:15.1956174Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8096
2022-09-27T16:14:15.1988653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8097
2022-09-27T16:14:16.4251359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:16.4251758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:16.4254431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:16.4254947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:16.4582295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:16.4582645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:16.4586755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:16.4587121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:16.5479544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:16.5958256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:16.9027359Z skip: Need at least 4 CUDA devices (1.712s)
2022-09-27T16:14:16.9043986Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:16.9089901Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8118
2022-09-27T16:14:18.1221051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:18.1221417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:18.1223714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:18.1224497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:18.2442694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:18.2449676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:18.2450326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:18.5121565Z ok (1.609s)
2022-09-27T16:14:18.5131991Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:18.5177846Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8132
2022-09-27T16:14:19.7320303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:19.7320953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:19.7323921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:19.7324522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:19.8536761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:19.8544199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:19.8545447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:20.1208772Z ok (1.609s)
2022-09-27T16:14:20.1216449Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:20.1261987Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8146
2022-09-27T16:14:21.3442536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:21.3442902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:21.3445160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:21.3445650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:21.4658928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:21.4666211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:21.4666881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:21.7292088Z ok (1.608s)
2022-09-27T16:14:21.7314653Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:21.7356350Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8160
2022-09-27T16:14:22.9766695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:22.9767086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:22.9769518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:22.9769869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:23.0991912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:23.0999135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:23.1000081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:23.3386792Z ok (1.609s)
2022-09-27T16:14:23.3396245Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:23.3440386Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8174
2022-09-27T16:14:24.5924479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:24.5924885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:24.5928038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:24.5928404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:24.7149517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:24.7156235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:24.7157218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:24.9472017Z ok (1.608s)
2022-09-27T16:14:24.9484331Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:24.9533394Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8188
2022-09-27T16:14:26.1332923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:26.1333353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:26.1335523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:26.1335916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:26.2560205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:26.2567506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:26.2568209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:26.4563542Z ok (1.509s)
2022-09-27T16:14:26.4571561Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:26.4614983Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8202
2022-09-27T16:14:27.6394664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:27.6395018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:27.6397458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:27.6397965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:27.7615870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:27.7622948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:27.7623786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:27.9646323Z ok (1.508s)
2022-09-27T16:14:27.9654639Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:14:27.9700070Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8216
2022-09-27T16:14:29.1553943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:29.1554315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:29.1557092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:29.1557682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:29.2774912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:29.2781710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:29.2782564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:14:29.5731808Z ok (1.608s)
2022-09-27T16:14:29.5731989Z 
2022-09-27T16:14:29.5732290Z ----------------------------------------------------------------------
2022-09-27T16:14:29.5732599Z Ran 42 tests in 38.020s
2022-09-27T16:14:29.5732719Z 
2022-09-27T16:14:29.5732798Z OK (skipped=32)
2022-09-27T16:14:29.5732908Z 
2022-09-27T16:14:29.5732982Z Generating XML reports...
2022-09-27T16:14:29.5801853Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220927161351.xml
2022-09-27T16:14:29.5810567Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220927161351.xml
2022-09-27T16:14:29.8614121Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-09-27 16:14:29.860969]
2022-09-27T16:14:29.8614938Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:14:29.861052]
2022-09-27T16:14:31.0624376Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2022-09-27T16:14:31.0638201Z 
2022-09-27T16:14:31.0638713Z Running tests...
2022-09-27T16:14:31.0639109Z ----------------------------------------------------------------------
2022-09-27T16:14:31.0646586Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0653681Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0661014Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0668104Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0674984Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0682447Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:31.0696007Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:31.3095237Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8241
2022-09-27T16:14:31.3120710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8242
2022-09-27T16:14:31.3147479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8243
2022-09-27T16:14:31.3175879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8244
2022-09-27T16:14:32.4860250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.4860867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.4864584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.4865224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.4992481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.4993081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.4996761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.4997365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.5008843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.5009442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.5013744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.5014599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.5137519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.5138410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.5142722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.5143448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.6128442Z dist init r=1, world=4
2022-09-27T16:14:32.6260254Z dist init r=2, world=4
2022-09-27T16:14:32.6262518Z dist init r=3, world=4
2022-09-27T16:14:32.6417897Z dist init r=0, world=4
2022-09-27T16:14:32.6670371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:32.6771101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:32.6873728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:32.6874737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:32.6875755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:32.6876511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:32.6877167Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:32.6877727Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:32.6881874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:32.6883442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:32.6884070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:32.6884436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:33.0225599Z skip: Need at least 2 CUDA devices (1.954s)
2022-09-27T16:14:33.0239800Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:33.0285236Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8297
2022-09-27T16:14:33.0318762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8298
2022-09-27T16:14:33.0348212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8299
2022-09-27T16:14:33.0375989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8300
2022-09-27T16:14:34.2177374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:34.2178048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:34.2180408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:34.2181043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:34.2607137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:34.2607727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:34.2610246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:34.2610848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:34.2791021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:34.2791719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:34.2796343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:34.2797006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:34.2931260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:34.2931890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:34.2935764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:34.2936412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:34.3451542Z dist init r=0, world=4
2022-09-27T16:14:34.3858080Z dist init r=1, world=4
2022-09-27T16:14:34.4085328Z dist init r=2, world=4
2022-09-27T16:14:34.4182105Z dist init r=3, world=4
2022-09-27T16:14:34.4290734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:34.4395814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:34.4396532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:34.4397229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:34.4397856Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:34.4398383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:34.4398925Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:34.4494600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:34.4504400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:34.4505166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:34.4505873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:34.4506517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:34.7418400Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:34.7432144Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:14:34.7478503Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8353
2022-09-27T16:14:34.7511448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8354
2022-09-27T16:14:34.7539414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8355
2022-09-27T16:14:34.7567411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8356
2022-09-27T16:14:35.9028978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:35.9029565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:35.9032616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:35.9033209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:35.9504963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:35.9505639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:35.9507840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:35.9508656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:35.9994488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:35.9995099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:36.0000183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:36.0000951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:36.0249749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:36.0250384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:36.0254320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:36.0254983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:36.0305934Z dist init r=3, world=4
2022-09-27T16:14:36.0793826Z dist init r=0, world=4
2022-09-27T16:14:36.1396111Z dist init r=2, world=4
2022-09-27T16:14:36.1547430Z dist init r=1, world=4
2022-09-27T16:14:36.1909026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:36.1909411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:36.2011488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:36.2012119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:36.2012780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:36.2013360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:36.2013883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:36.2014402Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:36.2117956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:36.2118340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:36.2118712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:36.2119258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:36.5611821Z skip: Need at least 2 CUDA devices (1.819s)
2022-09-27T16:14:36.5629018Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:36.5674754Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8409
2022-09-27T16:14:36.5707732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8410
2022-09-27T16:14:36.5736290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8411
2022-09-27T16:14:36.5764909Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8412
2022-09-27T16:14:37.7147578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:37.7148506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:37.7151192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:37.7151970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:37.7755723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:37.7756410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:37.7760580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:37.7761204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:37.8096310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:37.8097009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:37.8100064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:37.8100693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:37.8168997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:37.8169617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:37.8173680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:37.8174296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:37.8418621Z dist init r=1, world=4
2022-09-27T16:14:37.9019557Z dist init r=2, world=4
2022-09-27T16:14:37.9344351Z dist init r=0, world=4
2022-09-27T16:14:37.9447188Z dist init r=3, world=4
2022-09-27T16:14:37.9555099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:37.9733552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:37.9734052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:37.9734471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:37.9735074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:37.9735608Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:37.9736128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:37.9758341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:37.9841056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:37.9842275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:37.9842859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:37.9864589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:38.2805134Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:38.2822379Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:38.2867170Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8465
2022-09-27T16:14:38.2900076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8466
2022-09-27T16:14:38.2927465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8467
2022-09-27T16:14:38.2955532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8468
2022-09-27T16:14:39.4613062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:39.4613725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:39.4616626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:39.4617241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:39.4670307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:39.4670881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:39.4675361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:39.4675997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:39.4689791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:39.4690399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:39.4696484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:39.4697110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:39.5295417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:39.5296049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:39.5299661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:39.5300351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:39.5880913Z dist init r=1, world=4
2022-09-27T16:14:39.5932464Z dist init r=3, world=4
2022-09-27T16:14:39.5989406Z dist init r=2, world=4
2022-09-27T16:14:39.6534897Z dist init r=0, world=4
2022-09-27T16:14:39.6701310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:39.6893604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:39.6995326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:39.6996033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:39.6997029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:39.6997599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:39.6998124Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:39.7004780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:39.7103373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:39.7104588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:39.7144690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:39.7145054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:39.9997080Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:40.0014267Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:14:40.0057718Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8521
2022-09-27T16:14:40.0090019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8522
2022-09-27T16:14:40.0116886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8523
2022-09-27T16:14:40.0144721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8524
2022-09-27T16:14:41.1281352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.1281956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.1285742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.1953787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.1954841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.1955431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.1958359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.1958937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.2091213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.2091815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.2095518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.2096145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.2274968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.2275609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.2279870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.2280496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.2561290Z dist init r=2, world=4
2022-09-27T16:14:41.3233424Z dist init r=0, world=4
2022-09-27T16:14:41.3511421Z dist init r=3, world=4
2022-09-27T16:14:41.3606224Z dist init r=1, world=4
2022-09-27T16:14:41.3917099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:41.4019119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:41.4121008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:41.4122147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:41.4122898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:41.4123808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:41.4124607Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:41.4125441Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:41.4127454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:41.4128338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:41.4128889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:41.4129515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:41.7187098Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:41.7205128Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:41.7250961Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8577
2022-09-27T16:14:41.7285064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8578
2022-09-27T16:14:41.7313942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8579
2022-09-27T16:14:41.7342198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8580
2022-09-27T16:14:42.9217429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:42.9218051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:42.9220455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:42.9221071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:42.9221840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:42.9222459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:42.9225574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:42.9226158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:42.9239273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:42.9239849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:42.9245131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:42.9245782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:42.9459019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:42.9459628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:42.9462908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:42.9463606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:43.0487119Z dist init r=0, world=4
2022-09-27T16:14:43.0490964Z dist init r=2, world=4
2022-09-27T16:14:43.0533458Z dist init r=1, world=4
2022-09-27T16:14:43.0706831Z dist init r=3, world=4
2022-09-27T16:14:43.0814510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:43.0915834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:43.1017368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:43.1017963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:43.1018896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:43.1019686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:43.1020480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:43.1023362Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:43.1025138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:43.1025710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:43.1026231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:43.1027754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:43.4384422Z skip: Need at least 2 CUDA devices (1.720s)
2022-09-27T16:14:43.4403021Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:43.4446786Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8633
2022-09-27T16:14:43.4479888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8634
2022-09-27T16:14:43.4508853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8635
2022-09-27T16:14:43.4537528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8636
2022-09-27T16:14:44.6107461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.6108120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.6110934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.6111510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.6295176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.6295855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.6300227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.6300867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.6428469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.6429121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.6433770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.6434390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.6534251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.6534886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.6538797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.6539433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.7375616Z dist init r=0, world=4
2022-09-27T16:14:44.7592854Z dist init r=2, world=4
2022-09-27T16:14:44.7677367Z dist init r=3, world=4
2022-09-27T16:14:44.7778721Z dist init r=1, world=4
2022-09-27T16:14:44.8004450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:44.8105756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:44.8106943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:44.8108521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:44.8109651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:44.8110270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:44.8111145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:44.8111963Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:44.8113192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:44.8113771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:44.8114268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:44.8114628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:45.1578642Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:45.1596610Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:14:45.1643460Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8689
2022-09-27T16:14:45.1676805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8690
2022-09-27T16:14:45.1705016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8691
2022-09-27T16:14:45.1733164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8692
2022-09-27T16:14:46.3621888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:46.3622445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:46.3626029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:46.3626707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:46.3744613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:46.3745204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:46.3749365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:46.3749999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:46.3789039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:46.3789661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:46.3794334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:46.3794945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:46.4060986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:46.4061616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:46.4065616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:46.4066269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:46.4884545Z dist init r=2, world=4
2022-09-27T16:14:46.5043447Z dist init r=3, world=4
2022-09-27T16:14:46.5174972Z dist init r=1, world=4
2022-09-27T16:14:46.5386016Z dist init r=0, world=4
2022-09-27T16:14:46.5685720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:46.5888332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:46.5888968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:46.5890064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:46.5890667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:46.5891448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:46.5892113Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:46.5892636Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:46.5995495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:46.5996064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:46.5996610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:46.5997131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:46.8774269Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:46.8781047Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8787071Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8792691Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8798306Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8803894Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8809296Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:14:46.8816099Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:46.8860197Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8745
2022-09-27T16:14:46.8892397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8746
2022-09-27T16:14:46.8921057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8747
2022-09-27T16:14:46.8949777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8748
2022-09-27T16:14:48.0707239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.0707873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.0710013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.0710584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.0889774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.0890657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.0894195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.0895009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.0952001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.0952611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.0956477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.0957085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.1514901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.1515564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.1518802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.1519400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.1966869Z dist init r=0, world=4
2022-09-27T16:14:48.2153622Z dist init r=3, world=4
2022-09-27T16:14:48.2252558Z dist init r=2, world=4
2022-09-27T16:14:48.2760769Z dist init r=1, world=4
2022-09-27T16:14:48.3071969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:48.3172696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:48.3275153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:48.3276027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:48.3276590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:48.3277131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:48.3277710Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:48.3278230Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:48.3380360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:48.3381026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:48.3381466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:48.3381906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:48.6993109Z skip: Need at least 2 CUDA devices (1.818s)
2022-09-27T16:14:48.7001335Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:48.7046587Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8801
2022-09-27T16:14:48.7079466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8802
2022-09-27T16:14:48.7107464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8803
2022-09-27T16:14:48.7143278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8804
2022-09-27T16:14:49.8020659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:49.8021315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:49.8024780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:49.8025421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:49.8539698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:49.8540329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:49.8544100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:49.8544796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:49.8639843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:49.8640460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:49.8644628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:49.8645227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:49.9182040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:49.9182701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:49.9186345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:49.9187395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:49.9287103Z dist init r=1, world=4
2022-09-27T16:14:49.9806740Z dist init r=2, world=4
2022-09-27T16:14:49.9887067Z dist init r=0, world=4
2022-09-27T16:14:50.0450223Z dist init r=3, world=4
2022-09-27T16:14:50.0557706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:50.0604585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:50.0605143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:50.0605675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:50.0606284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:50.0606819Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:50.0607342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:50.0660031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:50.0712145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:50.0712839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:50.0713391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:50.0713939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:50.4185057Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:50.4194379Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:14:50.4240713Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8857
2022-09-27T16:14:50.4273522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8858
2022-09-27T16:14:50.4302224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8859
2022-09-27T16:14:50.4331125Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8860
2022-09-27T16:14:51.6133352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.6134275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.6137563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.6138205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.6333882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.6334494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.6338297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.6338958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.6654522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.6655131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.6658330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.6658958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.6944762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.6945392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.6948988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.6949637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.7448957Z dist init r=2, world=4
2022-09-27T16:14:51.7755043Z dist init r=3, world=4
2022-09-27T16:14:51.7949894Z dist init r=0, world=4
2022-09-27T16:14:51.8231832Z dist init r=1, world=4
2022-09-27T16:14:51.8365383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:51.8562547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:51.8563781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:51.8564517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:51.8564904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:51.8565417Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:51.8566031Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:51.8568946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:51.8670925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:51.8671617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:51.8671989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:51.8672326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:52.1374026Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:52.1385237Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:52.1430764Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8913
2022-09-27T16:14:52.1461761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8914
2022-09-27T16:14:52.1490035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8915
2022-09-27T16:14:52.1518182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8916
2022-09-27T16:14:53.3308165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.3308683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.3312133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.3312638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.3324715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.3325254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.3329461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.3330022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.3714607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.3715259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.3716056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.3716675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.3717444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.3717990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.3719464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.3720056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.4570055Z dist init r=2, world=4
2022-09-27T16:14:53.4625617Z dist init r=0, world=4
2022-09-27T16:14:53.4962315Z dist init r=3, world=4
2022-09-27T16:14:53.4965532Z dist init r=1, world=4
2022-09-27T16:14:53.5171717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:53.5271365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:53.5339109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:53.5339625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:53.5340449Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:53.5341009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:53.5374111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:53.5375281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:53.5479554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:53.5480098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:53.5480826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:53.5481994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:53.8559003Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:14:53.8571088Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:14:53.8615374Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8969
2022-09-27T16:14:53.8647774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8970
2022-09-27T16:14:53.8676584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8971
2022-09-27T16:14:53.8713702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8972
2022-09-27T16:14:55.0526448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:55.0527081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:55.0530593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:55.0531223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:55.0585717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:55.0586412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:55.0590304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:55.0590999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:55.0780336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:55.0780981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:55.0784160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:55.0784812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:55.0879761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:55.0880379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:55.0884025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:55.0884649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:55.1820346Z dist init r=1, world=4
2022-09-27T16:14:55.1844382Z dist init r=3, world=4
2022-09-27T16:14:55.2041824Z dist init r=0, world=4
2022-09-27T16:14:55.2123874Z dist init r=2, world=4
2022-09-27T16:14:55.2253512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:55.2355627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:55.2460616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:55.2461122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:55.2462016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:55.2462849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:55.2463677Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:55.2464928Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:55.2465578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:55.2466216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:55.2467803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:55.2468372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:55.5755196Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:14:55.5767161Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:14:55.5811652Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9025
2022-09-27T16:14:55.5844115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9026
2022-09-27T16:14:55.5871154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9027
2022-09-27T16:14:55.5899264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9028
2022-09-27T16:14:56.7596015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:56.7596633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:56.7599914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:56.7600511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:56.7766909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:56.7767548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:56.7771313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:56.7771954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:56.7776749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:56.7777281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:56.7781894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:56.7782543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:56.7896578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:56.7897170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:56.7901863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:56.7902446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:56.8854899Z dist init r=0, world=4
2022-09-27T16:14:56.9047224Z dist init r=1, world=4
2022-09-27T16:14:56.9140818Z dist init r=2, world=4
2022-09-27T16:14:56.9186149Z dist init r=3, world=4
2022-09-27T16:14:56.9358612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:56.9359325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:56.9359913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:14:56.9360473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:14:56.9361303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:56.9362270Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:56.9363125Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:56.9363796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:14:56.9366237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:14:56.9366717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:56.9367253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:56.9367812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:14:57.2941294Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:14:57.2949478Z 
2022-09-27T16:14:57.2950094Z ----------------------------------------------------------------------
2022-09-27T16:14:57.2950431Z Ran 27 tests in 26.230s
2022-09-27T16:14:57.2950560Z 
2022-09-27T16:14:57.2950638Z OK (skipped=27)
2022-09-27T16:14:57.2950750Z 
2022-09-27T16:14:57.2950836Z Generating XML reports...
2022-09-27T16:14:57.3004566Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220927161431.xml
2022-09-27T16:14:57.5809941Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-09-27 16:14:57.580551]
2022-09-27T16:14:57.5810548Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:14:57.580629]
2022-09-27T16:14:58.7779181Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-09-27T16:14:58.7790257Z 
2022-09-27T16:14:58.7790384Z Running tests...
2022-09-27T16:14:58.7790884Z ----------------------------------------------------------------------
2022-09-27T16:14:58.7802215Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-09-27T16:14:59.0170914Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9092
2022-09-27T16:14:59.0197488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9093
2022-09-27T16:14:59.0225280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9094
2022-09-27T16:14:59.0254188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9095
2022-09-27T16:15:00.2304553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.2308271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.2309091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.2309700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.2691573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.2692178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.2695217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.2695855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.2949187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.2949846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.2954772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.2955422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.2994973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.2995565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.2999594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.3000240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.3579561Z dist init r=3, world=4
2022-09-27T16:15:00.3947031Z dist init r=2, world=4
2022-09-27T16:15:00.4243376Z dist init r=1, world=4
2022-09-27T16:15:00.4247193Z dist init r=0, world=4
2022-09-27T16:15:00.4391680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:00.4593142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:00.4694573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:00.4695236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:00.4696220Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:00.4697061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:00.4697583Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:00.4698105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:00.4802564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:00.4803266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:00.4803804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:00.4804334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:00.8301571Z skip: Need at least 2 CUDA devices (2.051s)
2022-09-27T16:15:00.8314082Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:00.8359900Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9148
2022-09-27T16:15:00.8392477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9149
2022-09-27T16:15:00.8420280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9150
2022-09-27T16:15:00.8448710Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9151
2022-09-27T16:15:02.0196795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:02.0197446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:02.0199177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:02.0199746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:02.0200518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:02.0201135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:02.0205157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:02.0205786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:02.0337713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:02.0338378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:02.0343525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:02.0344397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:02.0420606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:02.0421217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:02.0425895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:02.0426524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:02.1486482Z dist init r=1, world=4
2022-09-27T16:15:02.1491902Z dist init r=0, world=4
2022-09-27T16:15:02.1657875Z dist init r=2, world=4
2022-09-27T16:15:02.1839643Z dist init r=3, world=4
2022-09-27T16:15:02.1967453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:02.2004380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:02.2004978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:02.2005361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:02.2005964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:02.2006514Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:02.2007051Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:02.2070092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:02.2111727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:02.2112414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:02.2113098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:02.2113760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:02.5492723Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:15:02.5504277Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-09-27T16:15:02.5550098Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9204
2022-09-27T16:15:02.5582661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9205
2022-09-27T16:15:02.5611027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9206
2022-09-27T16:15:02.5640215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9207
2022-09-27T16:15:03.7285880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.7286525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.7289672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.7290311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.7424286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.7424950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.7429036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.7430057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.7430910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.7431485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.7436884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.7437531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.8188965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.8189605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.8191365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.8192031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.8552249Z dist init r=0, world=4
2022-09-27T16:15:03.8694269Z dist init r=1, world=4
2022-09-27T16:15:03.8723565Z dist init r=2, world=4
2022-09-27T16:15:03.9433858Z dist init r=3, world=4
2022-09-27T16:15:03.9638742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:03.9639334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:03.9641239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:03.9641878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:03.9642805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:03.9643454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:03.9644121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:03.9644698Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:03.9648962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:03.9649547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:03.9650108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:03.9650639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:04.2680698Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:15:04.2692211Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:04.2736698Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9260
2022-09-27T16:15:04.2770426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9261
2022-09-27T16:15:04.2799070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9262
2022-09-27T16:15:04.2834795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9263
2022-09-27T16:15:05.4248640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.4249284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.4253367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.4254028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:05.4534168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.4534675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.4537999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.4538506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:05.4844769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.4845433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.4848496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.4849064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:05.4902936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.4903582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.4907770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.4908413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:05.5528279Z dist init r=0, world=4
2022-09-27T16:15:05.5796945Z dist init r=2, world=4
2022-09-27T16:15:05.6093400Z dist init r=3, world=4
2022-09-27T16:15:05.6182027Z dist init r=1, world=4
2022-09-27T16:15:05.6541685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:05.6642741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:05.6745537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:05.6746138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:05.6747067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:05.6747633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:05.6748256Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:05.6748959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:05.6753652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:05.6754288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:05.6754855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:05.6755216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:05.9875479Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:15:05.9886708Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:05.9929385Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9316
2022-09-27T16:15:05.9960840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9317
2022-09-27T16:15:05.9988174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9318
2022-09-27T16:15:06.0015909Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9319
2022-09-27T16:15:07.1698985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.1699583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.1702301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.1702932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.2180141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.2180804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.2183450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.2184289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.2253722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.2254383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.2258849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.2259515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.2428878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.2429553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.2433806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.2434419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.2985748Z dist init r=2, world=4
2022-09-27T16:15:07.3451971Z dist init r=1, world=4
2022-09-27T16:15:07.3513656Z dist init r=0, world=4
2022-09-27T16:15:07.3821523Z dist init r=3, world=4
2022-09-27T16:15:07.3928672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:07.4026440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:07.4027278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:07.4028271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:07.4029005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:07.4029642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:07.4030280Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:07.4031239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:07.4134881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:07.4135535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:07.4136086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:07.4136716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:07.7056803Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:15:07.7068263Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-09-27T16:15:07.7112068Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9372
2022-09-27T16:15:07.7144615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9373
2022-09-27T16:15:07.7171697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9374
2022-09-27T16:15:07.7199823Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9375
2022-09-27T16:15:08.8448105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:08.8448782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:08.8452685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:08.8453304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:08.8856737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:08.8857395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:08.8860814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:08.8861485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:08.8904391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:08.8905058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:08.8908897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:08.8909554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:08.9047463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:08.9048063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:08.9051797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:08.9052477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:08.9720234Z dist init r=0, world=4
2022-09-27T16:15:09.0153752Z dist init r=1, world=4
2022-09-27T16:15:09.0160070Z dist init r=3, world=4
2022-09-27T16:15:09.0295389Z dist init r=2, world=4
2022-09-27T16:15:09.0469402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:09.0532767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:09.0634431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:09.0634942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:09.0635679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:09.0636200Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:09.0636905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:09.0673121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:09.0742667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:09.0743282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:09.0743661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:09.0744278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:09.4240276Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:15:09.4251448Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:09.4286309Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9428
2022-09-27T16:15:09.4313666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9429
2022-09-27T16:15:09.4340937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9430
2022-09-27T16:15:09.4367831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9431
2022-09-27T16:15:10.6139049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:10.6139660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:10.6143010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:10.6143642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:10.6154370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:10.6154991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:10.6159349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:10.6159987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:10.6212469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:10.6213049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:10.6217167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:10.6217793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:10.6526014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:10.6526644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:10.6530097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:10.6530733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:10.7406119Z dist init r=1, world=4
2022-09-27T16:15:10.7458717Z dist init r=2, world=4
2022-09-27T16:15:10.7472460Z dist init r=3, world=4
2022-09-27T16:15:10.7778390Z dist init r=0, world=4
2022-09-27T16:15:10.8070537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:10.8173292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:10.8174232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:10.8174852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:10.8175917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:10.8177014Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:10.8177833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:10.8178681Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:10.8179923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:10.8180470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:10.8181245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:10.8185609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:11.1408375Z skip: Need at least 2 CUDA devices (1.717s)
2022-09-27T16:15:11.1419591Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:11.1466566Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9484
2022-09-27T16:15:11.1501117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9485
2022-09-27T16:15:11.1528486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9486
2022-09-27T16:15:11.1556452Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9487
2022-09-27T16:15:12.3353342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.3353885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.3357393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.3358041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.3615179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.3615890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.3619094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.3619810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.3797371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.3797988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.3802443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.3803121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.3844794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.3845420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.3850061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.3850694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.4622866Z dist init r=0, world=4
2022-09-27T16:15:12.4863302Z dist init r=1, world=4
2022-09-27T16:15:12.5082683Z dist init r=2, world=4
2022-09-27T16:15:12.5181168Z dist init r=3, world=4
2022-09-27T16:15:12.5289632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:12.5438635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:12.5439287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:12.5439994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:12.5441069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:12.5441699Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:12.5442231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:12.5492633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:12.5546853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:12.5548552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:12.5549092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:12.5550020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:12.8598552Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:15:12.8609766Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-09-27T16:15:12.8654611Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9540
2022-09-27T16:15:12.8687497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9541
2022-09-27T16:15:12.8715144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9542
2022-09-27T16:15:12.8742803Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9543
2022-09-27T16:15:14.0456834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:14.0457391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:14.0460485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:14.0461060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:14.0741463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:14.0742084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:14.0745550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:14.0746339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:14.0770835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:14.0771374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:14.0776370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:14.0776964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:14.0883112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:14.0883734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:14.0888139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:14.0888943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:14.1724160Z dist init r=0, world=4
2022-09-27T16:15:14.1999750Z dist init r=1, world=4
2022-09-27T16:15:14.2070814Z dist init r=2, world=4
2022-09-27T16:15:14.2139974Z dist init r=3, world=4
2022-09-27T16:15:14.2309869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:14.2411193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:14.2513646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:14.2514467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:14.2515634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:14.2516385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:14.2517173Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:14.2517984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:14.2520716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:14.2521394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:14.2521876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:14.2523458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:14.5784410Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:15:14.5797340Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:14.5844447Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9596
2022-09-27T16:15:14.5876889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9597
2022-09-27T16:15:14.5906876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9598
2022-09-27T16:15:14.5948371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9599
2022-09-27T16:15:15.7408297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.7408959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.7412723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.7413363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:15.7414135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.7414710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.7418823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.7419446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:15.7576334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.7576929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.7580640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.7581464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:15.7863278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.7864064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.7867174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.7867826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:15.8681578Z dist init r=3, world=4
2022-09-27T16:15:15.8688035Z dist init r=1, world=4
2022-09-27T16:15:15.8837580Z dist init r=0, world=4
2022-09-27T16:15:15.9149005Z dist init r=2, world=4
2022-09-27T16:15:15.9448817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:15.9549956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:15.9653020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:15.9654152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:15.9655044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:15.9655619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:15.9656131Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:15.9656669Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:15.9660250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:15.9660793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:15.9661322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:15.9661846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:16.2990797Z skip: Need at least 2 CUDA devices (1.721s)
2022-09-27T16:15:16.3001957Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-09-27T16:15:16.3046484Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9652
2022-09-27T16:15:16.3079405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9653
2022-09-27T16:15:16.3106098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9654
2022-09-27T16:15:16.3134402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9655
2022-09-27T16:15:17.4806222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:17.4806845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:17.4810441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:17.4811036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:17.5059129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:17.5059890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:17.5062644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:17.5063353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:17.5222971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:17.5223603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:17.5227809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:17.5228430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:17.5469784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:17.5470410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:17.5474501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:17.5475143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:17.6079856Z dist init r=0, world=4
2022-09-27T16:15:17.6309249Z dist init r=3, world=4
2022-09-27T16:15:17.6482138Z dist init r=2, world=4
2022-09-27T16:15:17.6747525Z dist init r=1, world=4
2022-09-27T16:15:17.6920727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:17.7020898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:17.7095518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:17.7096251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:17.7096900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:17.7097433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:17.7123794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:17.7124546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:17.7229940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:17.7230636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:17.7231145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:17.7231701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:18.0175647Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:15:18.0188253Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-09-27T16:15:18.0234109Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9708
2022-09-27T16:15:18.0266808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9709
2022-09-27T16:15:18.0294055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9710
2022-09-27T16:15:18.0321938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9711
2022-09-27T16:15:19.1879070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:19.1879957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:19.1882472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:19.1883248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:19.2032242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:19.2032846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:19.2037117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:19.2038308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:19.2039154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:19.2039758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:19.2043948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:19.2044573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:19.2886490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:19.2887146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:19.2890884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:19.2891557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:19.3153376Z dist init r=3, world=4
2022-09-27T16:15:19.3292843Z dist init r=2, world=4
2022-09-27T16:15:19.3298398Z dist init r=0, world=4
2022-09-27T16:15:19.4161441Z dist init r=1, world=4
2022-09-27T16:15:19.4368212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:19.4471284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:19.4573207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:19.4573955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:19.4574374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:19.4574861Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:19.4575393Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:19.4673163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:19.4680707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:19.4681318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:19.4681782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:19.4682280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:19.7363992Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:15:19.7364274Z 
2022-09-27T16:15:19.7364631Z ----------------------------------------------------------------------
2022-09-27T16:15:19.7364884Z Ran 12 tests in 20.957s
2022-09-27T16:15:19.7365003Z 
2022-09-27T16:15:19.7365080Z OK (skipped=12)
2022-09-27T16:15:19.7365194Z 
2022-09-27T16:15:19.7365283Z Generating XML reports...
2022-09-27T16:15:19.7413163Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220927161458.xml
2022-09-27T16:15:20.0266260Z Running distributed/test_pg_wrapper ... [2022-09-27 16:15:20.026168]
2022-09-27T16:15:20.0267121Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:15:20.026250]
2022-09-27T16:15:21.2176795Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:15:21.2177234Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:15:21.2178636Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-09-27T16:15:21.2179820Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2180132Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2180469Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2180822Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2181182Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2181504Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2181833Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2182176Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2182516Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:15:21.2183303Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-09-27T16:15:21.2184156Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:15:21.2184534Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:15:21.2184873Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:15:21.2185202Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:15:21.2185534Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:15:22.2514570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:22.2514950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:22.2517460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:22.2517812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:22.3845737Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:22.3855661Z 
2022-09-27T16:15:22.3855902Z Running tests...
2022-09-27T16:15:22.3856506Z ----------------------------------------------------------------------
2022-09-27T16:15:22.6186773Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9785
2022-09-27T16:15:22.6211744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9786
2022-09-27T16:15:22.6237581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9787
2022-09-27T16:15:22.6265281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9788
2022-09-27T16:15:23.8505260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:23.8505885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:23.8507143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:23.8507721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.8629246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:23.8629880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:23.8632524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:23.8633124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.8883212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:23.8883844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:23.8886842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:23.8887499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.8890925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:23.8891508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:23.8894447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:23.8895082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.9989555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:24.0084591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:24.0236050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:24.0236433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:24.0392704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:24.0493593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:24.0596440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:24.0596970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:24.0597669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:24.0598277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:24.0598870Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:24.0599389Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:24.1318189Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:15:24.1318584Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:15:24.1418204Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:15:24.1519436Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:15:24.5315285Z ok (2.146s)
2022-09-27T16:15:24.5315553Z 
2022-09-27T16:15:24.5316085Z ----------------------------------------------------------------------
2022-09-27T16:15:24.5316365Z Ran 1 test in 2.146s
2022-09-27T16:15:24.5316484Z 
2022-09-27T16:15:24.5316551Z OK
2022-09-27T16:15:24.5316644Z 
2022-09-27T16:15:24.5316725Z Generating XML reports...
2022-09-27T16:15:24.5364007Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161522.xml
2022-09-27T16:15:25.8616915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:25.8617307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:25.8619472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:25.8619838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:25.9962737Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:25.9972299Z 
2022-09-27T16:15:25.9972375Z Running tests...
2022-09-27T16:15:25.9973039Z ----------------------------------------------------------------------
2022-09-27T16:15:26.2282371Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9876
2022-09-27T16:15:26.2307427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9877
2022-09-27T16:15:26.2333006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9878
2022-09-27T16:15:26.2360843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9879
2022-09-27T16:15:27.3813335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:27.3813925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:27.3815586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:27.3816145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:27.3852611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:27.3853262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:27.3855224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:27.3855804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:27.3856541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:27.3857099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:27.3859421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:27.3860068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:27.4484213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:27.4484870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:27.4486747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:27.4487401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:27.5152963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:27.5191509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:27.5191888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:27.5845290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:27.5965266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:27.5965872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:27.5966805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:27.5967266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:27.5967696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:27.5968204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:27.5968717Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:27.6067730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:28.1408431Z ok (2.143s)
2022-09-27T16:15:28.1408693Z 
2022-09-27T16:15:28.1409140Z ----------------------------------------------------------------------
2022-09-27T16:15:28.1409415Z Ran 1 test in 2.144s
2022-09-27T16:15:28.1409533Z 
2022-09-27T16:15:28.1409581Z OK
2022-09-27T16:15:28.1409675Z 
2022-09-27T16:15:28.1409769Z Generating XML reports...
2022-09-27T16:15:28.1458541Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161525.xml
2022-09-27T16:15:29.5004984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:29.5005581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:29.5007626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:29.5008198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:29.6337205Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:29.6347503Z 
2022-09-27T16:15:29.6347888Z Running tests...
2022-09-27T16:15:29.6348549Z ----------------------------------------------------------------------
2022-09-27T16:15:29.8664705Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9967
2022-09-27T16:15:29.8690496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9968
2022-09-27T16:15:29.8716541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9969
2022-09-27T16:15:29.8744745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9970
2022-09-27T16:15:31.0892974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.0893634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.0894824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.0895663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:31.0935017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.0935941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.0937228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.0937810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:31.0959463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.0960073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.0963315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.0963917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:31.1086597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.1087241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.1090232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.1090853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:31.2272059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:31.2300331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:31.2324167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:31.2656822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:31.5790123Z skip: Need at least 4 CUDA devices (1.944s)
2022-09-27T16:15:31.5790509Z 
2022-09-27T16:15:31.5790965Z ----------------------------------------------------------------------
2022-09-27T16:15:31.5791225Z Ran 1 test in 1.944s
2022-09-27T16:15:31.5791349Z 
2022-09-27T16:15:31.5791422Z OK (skipped=1)
2022-09-27T16:15:31.5791531Z 
2022-09-27T16:15:31.5791617Z Generating XML reports...
2022-09-27T16:15:31.5838664Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161529.xml
2022-09-27T16:15:32.9106761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:32.9107131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:32.9109247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:32.9109636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:33.0439263Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:33.0448774Z 
2022-09-27T16:15:33.0449011Z Running tests...
2022-09-27T16:15:33.0449402Z ----------------------------------------------------------------------
2022-09-27T16:15:33.2796547Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10022
2022-09-27T16:15:33.2822024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10023
2022-09-27T16:15:33.2848062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10024
2022-09-27T16:15:33.2875318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10025
2022-09-27T16:15:34.4267617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:34.4268511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:34.4270695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:34.4271449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:34.4367364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:34.4367979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:34.4370781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:34.4371395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:34.4525080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:34.4525666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:34.4528592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:34.4529247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:34.4667022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:34.4667624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:34.4670915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:34.4671565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:34.5619861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:34.5698426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:34.5859399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:34.6047909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:34.8919769Z skip: Need at least 4 CUDA devices (1.847s)
2022-09-27T16:15:34.8920029Z 
2022-09-27T16:15:34.8920346Z ----------------------------------------------------------------------
2022-09-27T16:15:34.8920665Z Ran 1 test in 1.847s
2022-09-27T16:15:34.8920820Z 
2022-09-27T16:15:34.8920896Z OK (skipped=1)
2022-09-27T16:15:34.8921004Z 
2022-09-27T16:15:34.8921099Z Generating XML reports...
2022-09-27T16:15:34.8968875Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161533.xml
2022-09-27T16:15:36.2253930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:36.2254307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:36.2256380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:36.2256787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:36.3624476Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:36.3633789Z 
2022-09-27T16:15:36.3633892Z Running tests...
2022-09-27T16:15:36.3634291Z ----------------------------------------------------------------------
2022-09-27T16:15:36.5966555Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10077
2022-09-27T16:15:36.5992698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10078
2022-09-27T16:15:36.6018878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10079
2022-09-27T16:15:36.6045749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10080
2022-09-27T16:15:37.7748705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.7749286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.7751282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.7751807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.8051885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.8052477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.8054843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.8055404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.8062046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.8062772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.8066084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.8066792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.8114765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.8115401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.8119144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.8119741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.9087476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:37.9388140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:37.9396641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:37.9480650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:37.9900797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:38.0102835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:38.0203998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:38.0204694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:38.0205381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:38.0205918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:38.0207898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:38.0208424Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:38.0918143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:38.1019329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:38.1019957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:15:38.1020930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-09-27T16:15:38.1021929Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:38.1022497Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:38.1122010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:38.1122614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:38.6096866Z ok (2.246s)
2022-09-27T16:15:38.6097125Z 
2022-09-27T16:15:38.6097549Z ----------------------------------------------------------------------
2022-09-27T16:15:38.6097806Z Ran 1 test in 2.246s
2022-09-27T16:15:38.6097923Z 
2022-09-27T16:15:38.6097986Z OK
2022-09-27T16:15:38.6098078Z 
2022-09-27T16:15:38.6098172Z Generating XML reports...
2022-09-27T16:15:38.6145775Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161536.xml
2022-09-27T16:15:39.9468655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:39.9469045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:39.9471047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:39.9471387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:40.0798432Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:40.0808713Z 
2022-09-27T16:15:40.0808804Z Running tests...
2022-09-27T16:15:40.0810123Z ----------------------------------------------------------------------
2022-09-27T16:15:40.3114614Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10180
2022-09-27T16:15:40.3140523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10181
2022-09-27T16:15:40.3166288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10182
2022-09-27T16:15:40.3195033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10183
2022-09-27T16:15:41.5434545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:41.5435201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:41.5437122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:41.5437682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:41.5440706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:41.5441389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:41.5445454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:41.5446261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:41.5692409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:41.5693118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:41.5695176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:41.5695802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:41.5939525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:41.5940188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:41.5942538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:41.5943170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:41.6779726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:41.6786354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:41.7036159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:41.7298542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:41.7598463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:41.7700853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:41.7701572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:41.7702264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:41.7702747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:41.7703252Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:41.7703785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:41.7802660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:42.3245835Z ok (2.243s)
2022-09-27T16:15:42.3246205Z 
2022-09-27T16:15:42.3246697Z ----------------------------------------------------------------------
2022-09-27T16:15:42.3247175Z Ran 1 test in 2.244s
2022-09-27T16:15:42.3247389Z 
2022-09-27T16:15:42.3247496Z OK
2022-09-27T16:15:42.3247585Z 
2022-09-27T16:15:42.3247677Z Generating XML reports...
2022-09-27T16:15:42.3295262Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161540.xml
2022-09-27T16:15:43.6789102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:43.6789491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:43.6791605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:43.6791970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:43.8115691Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:43.8124926Z 
2022-09-27T16:15:43.8125031Z Running tests...
2022-09-27T16:15:43.8125479Z ----------------------------------------------------------------------
2022-09-27T16:15:44.0437613Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10271
2022-09-27T16:15:44.0462907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10272
2022-09-27T16:15:44.0488957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10273
2022-09-27T16:15:44.0516594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10274
2022-09-27T16:15:45.1957765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.1958301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.1959405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.1960317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.2155431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.2156330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.2157678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.2158243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.2598025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.2598649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.2600124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.2600744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.2629502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.2630119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.2633127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.2633748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.3314011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:45.3496979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:45.3978105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:45.4218685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:45.7562382Z skip: Need at least 4 CUDA devices (1.943s)
2022-09-27T16:15:45.7562729Z 
2022-09-27T16:15:45.7563274Z ----------------------------------------------------------------------
2022-09-27T16:15:45.7563741Z Ran 1 test in 1.944s
2022-09-27T16:15:45.7563959Z 
2022-09-27T16:15:45.7564095Z OK (skipped=1)
2022-09-27T16:15:45.7564224Z 
2022-09-27T16:15:45.7564321Z Generating XML reports...
2022-09-27T16:15:45.7609309Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161543.xml
2022-09-27T16:15:47.0833141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:47.0833518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:47.0835454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:47.0835822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:47.2160965Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:47.2170191Z 
2022-09-27T16:15:47.2170291Z Running tests...
2022-09-27T16:15:47.2171239Z ----------------------------------------------------------------------
2022-09-27T16:15:47.4481648Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10326
2022-09-27T16:15:47.4506999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10327
2022-09-27T16:15:47.4532902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10328
2022-09-27T16:15:47.4559950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10329
2022-09-27T16:15:48.6029512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:48.6030445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:48.6032429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:48.6033186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:48.6164539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:48.6166866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:48.6167680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:48.6168310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:48.6301026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:48.6301644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:48.6305033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:48.6305759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:48.6338978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:48.6339600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:48.6343469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:48.6344365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:48.7374523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:48.7671692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:48.7675213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:48.7690289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:49.0605350Z skip: Need at least 4 CUDA devices (1.843s)
2022-09-27T16:15:49.0605636Z 
2022-09-27T16:15:49.0606189Z ----------------------------------------------------------------------
2022-09-27T16:15:49.0606476Z Ran 1 test in 1.843s
2022-09-27T16:15:49.0606579Z 
2022-09-27T16:15:49.0606652Z OK (skipped=1)
2022-09-27T16:15:49.0606761Z 
2022-09-27T16:15:49.0606848Z Generating XML reports...
2022-09-27T16:15:49.0653070Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161547.xml
2022-09-27T16:15:50.4126739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:50.4127108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:50.4129425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:50.4129875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:50.5452178Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:50.5461632Z 
2022-09-27T16:15:50.5462014Z Running tests...
2022-09-27T16:15:50.5462436Z ----------------------------------------------------------------------
2022-09-27T16:15:50.7787527Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10381
2022-09-27T16:15:50.7813087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10382
2022-09-27T16:15:50.7838654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10383
2022-09-27T16:15:50.7865562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10384
2022-09-27T16:15:51.8801768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:51.8802414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:51.8803612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:51.8804222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:51.9475643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:51.9476264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:51.9477655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:51.9478211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:51.9566028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:51.9566656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:51.9569519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:51.9759548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:51.9760491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:51.9761109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:51.9762782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:51.9763424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:52.0147058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:15:52.0819210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:52.0898736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:52.1124109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:15:52.1737367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:52.1839640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:52.1840338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:15:52.1840913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:15:52.1841950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:52.1842926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:52.1843744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:52.1845442Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:15:52.2757867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:52.2858154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:52.2959808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-09-27T16:15:52.2961370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:15:52.2962815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:52.2963767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:52.2964842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:52.2965479Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:15:52.7916383Z ok (2.245s)
2022-09-27T16:15:52.7916647Z 
2022-09-27T16:15:52.7917163Z ----------------------------------------------------------------------
2022-09-27T16:15:52.7917625Z Ran 1 test in 2.245s
2022-09-27T16:15:52.7917770Z 
2022-09-27T16:15:52.7917818Z OK
2022-09-27T16:15:52.7917913Z 
2022-09-27T16:15:52.7918008Z Generating XML reports...
2022-09-27T16:15:52.7963619Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161550.xml
2022-09-27T16:15:54.1644763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:54.1645278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:54.1647262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:54.1647870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:54.2978970Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:54.2988562Z 
2022-09-27T16:15:54.2988668Z Running tests...
2022-09-27T16:15:54.2989448Z ----------------------------------------------------------------------
2022-09-27T16:15:54.2993634Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:15:54.2994024Z 
2022-09-27T16:15:54.2994471Z ----------------------------------------------------------------------
2022-09-27T16:15:54.2994806Z Ran 1 test in 0.000s
2022-09-27T16:15:54.2994932Z 
2022-09-27T16:15:54.2995006Z OK (skipped=1)
2022-09-27T16:15:54.2995116Z 
2022-09-27T16:15:54.2995204Z Generating XML reports...
2022-09-27T16:15:54.3018893Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161554.xml
2022-09-27T16:15:55.5399231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:55.5399658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:55.5402183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:55.5402550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:55.6728244Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:55.6738350Z 
2022-09-27T16:15:55.6738858Z Running tests...
2022-09-27T16:15:55.6739268Z ----------------------------------------------------------------------
2022-09-27T16:15:55.6744444Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:15:55.6744987Z 
2022-09-27T16:15:55.6745244Z ----------------------------------------------------------------------
2022-09-27T16:15:55.6745492Z Ran 1 test in 0.001s
2022-09-27T16:15:55.6745607Z 
2022-09-27T16:15:55.6745680Z OK (skipped=1)
2022-09-27T16:15:55.6745775Z 
2022-09-27T16:15:55.6745863Z Generating XML reports...
2022-09-27T16:15:55.6769338Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161555.xml
2022-09-27T16:15:56.8912692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:56.8913143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:56.8914740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:56.8915179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:57.0264702Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:57.0274935Z 
2022-09-27T16:15:57.0275140Z Running tests...
2022-09-27T16:15:57.0275739Z ----------------------------------------------------------------------
2022-09-27T16:15:57.0281328Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:15:57.0281854Z 
2022-09-27T16:15:57.0282153Z ----------------------------------------------------------------------
2022-09-27T16:15:57.0282409Z Ran 1 test in 0.001s
2022-09-27T16:15:57.0282537Z 
2022-09-27T16:15:57.0282613Z OK (skipped=1)
2022-09-27T16:15:57.0282727Z 
2022-09-27T16:15:57.0282800Z Generating XML reports...
2022-09-27T16:15:57.0305922Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161557.xml
2022-09-27T16:15:58.2446367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:58.2446741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:58.2448398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:58.2448756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:58.3778194Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:58.3787795Z 
2022-09-27T16:15:58.3788257Z Running tests...
2022-09-27T16:15:58.3788669Z ----------------------------------------------------------------------
2022-09-27T16:15:58.3792841Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:15:58.3793451Z 
2022-09-27T16:15:58.3793767Z ----------------------------------------------------------------------
2022-09-27T16:15:58.3794014Z Ran 1 test in 0.000s
2022-09-27T16:15:58.3794127Z 
2022-09-27T16:15:58.3794198Z OK (skipped=1)
2022-09-27T16:15:58.3794304Z 
2022-09-27T16:15:58.3794376Z Generating XML reports...
2022-09-27T16:15:58.3817671Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161558.xml
2022-09-27T16:15:59.5999980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.6000360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.6002302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.6002661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:59.7331078Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:15:59.7340620Z 
2022-09-27T16:15:59.7340744Z Running tests...
2022-09-27T16:15:59.7341198Z ----------------------------------------------------------------------
2022-09-27T16:15:59.7346855Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:15:59.7347341Z 
2022-09-27T16:15:59.7347790Z ----------------------------------------------------------------------
2022-09-27T16:15:59.7348054Z Ran 1 test in 0.001s
2022-09-27T16:15:59.7348158Z 
2022-09-27T16:15:59.7348236Z OK (skipped=1)
2022-09-27T16:15:59.7348342Z 
2022-09-27T16:15:59.7348430Z Generating XML reports...
2022-09-27T16:15:59.7371548Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161559.xml
2022-09-27T16:16:00.1388571Z Running distributed/test_c10d_spawn_gloo ... [2022-09-27 16:16:00.138449]
2022-09-27T16:16:00.1389406Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:16:00.138533]
2022-09-27T16:16:01.3025387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6d950gf
2022-09-27T16:16:01.3026315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6d950gf/_remote_module_non_scriptable.py
2022-09-27T16:16:01.5229527Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-09-27T16:16:01.5230691Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:16:01.5231317Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:16:01.5231858Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:16:01.5232967Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_chunk_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]>
2022-09-27T16:16:01.5233603Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest)
2022-09-27T16:16:01.5233928Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest)
2022-09-27T16:16:01.5234240Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest)
2022-09-27T16:16:01.5234539Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest)
2022-09-27T16:16:01.5234814Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:16:01.5235056Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:16:01.5235940Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-09-27T16:16:01.5236823Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5237116Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5237402Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5237707Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5238003Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5238277Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5238564Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:01.5238846Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:16:02.5268647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:02.5269240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:02.5277696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:02.5278346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:02.6768438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgs1_8vs
2022-09-27T16:16:02.6769636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgs1_8vs/_remote_module_non_scriptable.py
2022-09-27T16:16:02.8985115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:02.9001215Z 
2022-09-27T16:16:02.9001719Z Running tests...
2022-09-27T16:16:02.9002147Z ----------------------------------------------------------------------
2022-09-27T16:16:02.9047750Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:02.9118066Z ok (0.012s)
2022-09-27T16:16:02.9118576Z 
2022-09-27T16:16:02.9119095Z ----------------------------------------------------------------------
2022-09-27T16:16:02.9119387Z Ran 1 test in 0.012s
2022-09-27T16:16:02.9119512Z 
2022-09-27T16:16:02.9119575Z OK
2022-09-27T16:16:02.9119667Z 
2022-09-27T16:16:02.9119757Z Generating XML reports...
2022-09-27T16:16:02.9142860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161602.xml
2022-09-27T16:16:04.2007313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:04.2007700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:04.2015626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:04.2016029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:04.3524509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgetuijff
2022-09-27T16:16:04.3526056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgetuijff/_remote_module_non_scriptable.py
2022-09-27T16:16:04.5740513Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:04.5756781Z 
2022-09-27T16:16:04.5756958Z Running tests...
2022-09-27T16:16:04.5757314Z ----------------------------------------------------------------------
2022-09-27T16:16:04.5762380Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s)
2022-09-27T16:16:04.5762697Z 
2022-09-27T16:16:04.5762928Z ----------------------------------------------------------------------
2022-09-27T16:16:04.5763171Z Ran 1 test in 0.001s
2022-09-27T16:16:04.5763285Z 
2022-09-27T16:16:04.5763358Z OK (skipped=1)
2022-09-27T16:16:04.5763452Z 
2022-09-27T16:16:04.5763536Z Generating XML reports...
2022-09-27T16:16:04.5788304Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161604.xml
2022-09-27T16:16:05.8370067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:05.8370453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:05.8378842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:05.8379234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:05.9877424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03or15l8
2022-09-27T16:16:05.9878097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03or15l8/_remote_module_non_scriptable.py
2022-09-27T16:16:06.2098237Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:06.2114345Z 
2022-09-27T16:16:06.2114561Z Running tests...
2022-09-27T16:16:06.2115142Z ----------------------------------------------------------------------
2022-09-27T16:16:06.2128940Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s)
2022-09-27T16:16:06.2129366Z 
2022-09-27T16:16:06.2130029Z ----------------------------------------------------------------------
2022-09-27T16:16:06.2130434Z Ran 1 test in 0.001s
2022-09-27T16:16:06.2130610Z 
2022-09-27T16:16:06.2130732Z OK (skipped=1)
2022-09-27T16:16:06.2131019Z 
2022-09-27T16:16:06.2131169Z Generating XML reports...
2022-09-27T16:16:06.2154694Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161606.xml
2022-09-27T16:16:07.4869065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:07.4869449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:07.4877371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:07.4877804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:07.6385536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2ehg1ts
2022-09-27T16:16:07.6387636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2ehg1ts/_remote_module_non_scriptable.py
2022-09-27T16:16:07.8600506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:07.8616542Z 
2022-09-27T16:16:07.8616677Z Running tests...
2022-09-27T16:16:07.8617271Z ----------------------------------------------------------------------
2022-09-27T16:16:07.8622605Z   test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-09-27T16:16:07.8623043Z 
2022-09-27T16:16:07.8623472Z ----------------------------------------------------------------------
2022-09-27T16:16:07.8624049Z Ran 1 test in 0.001s
2022-09-27T16:16:07.8624246Z 
2022-09-27T16:16:07.8624429Z OK (skipped=1)
2022-09-27T16:16:07.8624604Z 
2022-09-27T16:16:07.8624760Z Generating XML reports...
2022-09-27T16:16:07.8647837Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161607.xml
2022-09-27T16:16:09.1250019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:09.1250413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:09.1258282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:09.1258650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:09.2773112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk0vy83b
2022-09-27T16:16:09.2773801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk0vy83b/_remote_module_non_scriptable.py
2022-09-27T16:16:09.4977637Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:09.4993611Z 
2022-09-27T16:16:09.4993911Z Running tests...
2022-09-27T16:16:09.4994311Z ----------------------------------------------------------------------
2022-09-27T16:16:09.4998887Z   test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-09-27T16:16:09.4999506Z 
2022-09-27T16:16:09.4999892Z ----------------------------------------------------------------------
2022-09-27T16:16:09.5000319Z Ran 1 test in 0.001s
2022-09-27T16:16:09.5000545Z 
2022-09-27T16:16:09.5000682Z OK (skipped=1)
2022-09-27T16:16:09.5000851Z 
2022-09-27T16:16:09.5000935Z Generating XML reports...
2022-09-27T16:16:09.5023964Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161609.xml
2022-09-27T16:16:10.7631031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:10.7631405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:10.7639349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:10.7639726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:10.9159955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby6d5uzh
2022-09-27T16:16:10.9161130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby6d5uzh/_remote_module_non_scriptable.py
2022-09-27T16:16:11.1428298Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:11.1444130Z 
2022-09-27T16:16:11.1444397Z Running tests...
2022-09-27T16:16:11.1444752Z ----------------------------------------------------------------------
2022-09-27T16:16:11.1449347Z   test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-09-27T16:16:11.1449828Z 
2022-09-27T16:16:11.1450153Z ----------------------------------------------------------------------
2022-09-27T16:16:11.1450472Z Ran 1 test in 0.001s
2022-09-27T16:16:11.1450575Z 
2022-09-27T16:16:11.1450651Z OK (skipped=1)
2022-09-27T16:16:11.1450758Z 
2022-09-27T16:16:11.1450884Z Generating XML reports...
2022-09-27T16:16:11.1474592Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161611.xml
2022-09-27T16:16:12.4065275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:12.4065674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:12.4073213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:12.4073575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:12.5567059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdex2125v
2022-09-27T16:16:12.5568787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdex2125v/_remote_module_non_scriptable.py
2022-09-27T16:16:12.7804028Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:12.7820315Z 
2022-09-27T16:16:12.7820432Z Running tests...
2022-09-27T16:16:12.7821059Z ----------------------------------------------------------------------
2022-09-27T16:16:12.7825872Z   test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-09-27T16:16:12.7826261Z 
2022-09-27T16:16:12.7826694Z ----------------------------------------------------------------------
2022-09-27T16:16:12.7827007Z Ran 1 test in 0.001s
2022-09-27T16:16:12.7827120Z 
2022-09-27T16:16:12.7827193Z OK (skipped=1)
2022-09-27T16:16:12.7827302Z 
2022-09-27T16:16:12.7827386Z Generating XML reports...
2022-09-27T16:16:12.7850171Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161612.xml
2022-09-27T16:16:14.0217390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:14.0217767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:14.0225654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:14.0226004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:14.1722361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwxztzis
2022-09-27T16:16:14.1723537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwxztzis/_remote_module_non_scriptable.py
2022-09-27T16:16:14.3956632Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:14.3972408Z 
2022-09-27T16:16:14.3972534Z Running tests...
2022-09-27T16:16:14.3973193Z ----------------------------------------------------------------------
2022-09-27T16:16:14.4127233Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10617
2022-09-27T16:16:14.4155623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10618
2022-09-27T16:16:15.4474571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:15.4475254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:15.4568986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:15.4569556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:15.4809660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:15.4810282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:15.4895809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:15.4896496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:15.6064542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqa5es84q
2022-09-27T16:16:15.6065604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqa5es84q/_remote_module_non_scriptable.py
2022-09-27T16:16:15.6104012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkeub3n0
2022-09-27T16:16:15.6105345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkeub3n0/_remote_module_non_scriptable.py
2022-09-27T16:16:15.8353753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:15.8372028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:16.1192762Z skip: Need at least 2 CUDA devices (1.722s)
2022-09-27T16:16:16.1193071Z 
2022-09-27T16:16:16.1193389Z ----------------------------------------------------------------------
2022-09-27T16:16:16.1193671Z Ran 1 test in 1.722s
2022-09-27T16:16:16.1193773Z 
2022-09-27T16:16:16.1193848Z OK (skipped=1)
2022-09-27T16:16:16.1193957Z 
2022-09-27T16:16:16.1194042Z Generating XML reports...
2022-09-27T16:16:16.1225702Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161614.xml
2022-09-27T16:16:17.4308985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:17.4309367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:17.4317078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:17.4317444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:17.5823096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64ozwrrf
2022-09-27T16:16:17.5824158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64ozwrrf/_remote_module_non_scriptable.py
2022-09-27T16:16:17.8010392Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:17.8026094Z 
2022-09-27T16:16:17.8026214Z Running tests...
2022-09-27T16:16:17.8026827Z ----------------------------------------------------------------------
2022-09-27T16:16:17.8183656Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10650
2022-09-27T16:16:17.8212709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10651
2022-09-27T16:16:18.8583285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:18.8584061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:18.8893879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:18.8894483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:18.8959161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:18.8959658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:18.9267572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:18.9268114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:19.0112547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlmvsk4r
2022-09-27T16:16:19.0113172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlmvsk4r/_remote_module_non_scriptable.py
2022-09-27T16:16:19.0476678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xorp71_
2022-09-27T16:16:19.0478142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xorp71_/_remote_module_non_scriptable.py
2022-09-27T16:16:19.2361830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:19.2736142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:19.6252213Z skip: Need at least 2 CUDA devices (1.822s)
2022-09-27T16:16:19.6252484Z 
2022-09-27T16:16:19.6253009Z ----------------------------------------------------------------------
2022-09-27T16:16:19.6253353Z Ran 1 test in 1.823s
2022-09-27T16:16:19.6253469Z 
2022-09-27T16:16:19.6253541Z OK (skipped=1)
2022-09-27T16:16:19.6253652Z 
2022-09-27T16:16:19.6253740Z Generating XML reports...
2022-09-27T16:16:19.6288123Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161617.xml
2022-09-27T16:16:20.9445326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:20.9445715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:20.9454032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:20.9454457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:21.0961736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2ec2_dc
2022-09-27T16:16:21.0962449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2ec2_dc/_remote_module_non_scriptable.py
2022-09-27T16:16:21.3226670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:21.3242412Z 
2022-09-27T16:16:21.3242502Z Running tests...
2022-09-27T16:16:21.3242982Z ----------------------------------------------------------------------
2022-09-27T16:16:21.3403461Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10683
2022-09-27T16:16:21.3432289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10684
2022-09-27T16:16:22.3743578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:22.3744494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:22.3751393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:22.3752015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:22.4083235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:22.4084141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:22.4091102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:22.4091906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:22.5300597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe24s5dw
2022-09-27T16:16:22.5301291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe24s5dw/_remote_module_non_scriptable.py
2022-09-27T16:16:22.5315651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwyertaj
2022-09-27T16:16:22.5317166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwyertaj/_remote_module_non_scriptable.py
2022-09-27T16:16:22.7601344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:22.7604037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:23.0470503Z skip: Need at least 2 CUDA devices (1.722s)
2022-09-27T16:16:23.0470793Z 
2022-09-27T16:16:23.0471334Z ----------------------------------------------------------------------
2022-09-27T16:16:23.0471638Z Ran 1 test in 1.723s
2022-09-27T16:16:23.0471754Z 
2022-09-27T16:16:23.0471815Z OK (skipped=1)
2022-09-27T16:16:23.0471926Z 
2022-09-27T16:16:23.0472013Z Generating XML reports...
2022-09-27T16:16:23.0503523Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161621.xml
2022-09-27T16:16:24.4007477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:24.4008091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:24.4016407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:24.4017056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:24.5516452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzlzv4o_
2022-09-27T16:16:24.5517548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzlzv4o_/_remote_module_non_scriptable.py
2022-09-27T16:16:24.7754282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:24.7770040Z 
2022-09-27T16:16:24.7770349Z Running tests...
2022-09-27T16:16:24.7770983Z ----------------------------------------------------------------------
2022-09-27T16:16:24.7928901Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10716
2022-09-27T16:16:24.7957271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10717
2022-09-27T16:16:25.8078933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:25.8081694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:25.8224535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:25.8224915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:25.8388659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:25.8389052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:25.8555137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:25.8555506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:25.9614156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0tantzb
2022-09-27T16:16:25.9615121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0tantzb/_remote_module_non_scriptable.py
2022-09-27T16:16:25.9798386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7ebc2im
2022-09-27T16:16:25.9799212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7ebc2im/_remote_module_non_scriptable.py
2022-09-27T16:16:26.1868392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:26.2091014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:26.4995920Z skip: Need at least 2 CUDA devices (1.722s)
2022-09-27T16:16:26.4996235Z 
2022-09-27T16:16:26.4996661Z ----------------------------------------------------------------------
2022-09-27T16:16:26.4996914Z Ran 1 test in 1.722s
2022-09-27T16:16:26.4997030Z 
2022-09-27T16:16:26.4997092Z OK (skipped=1)
2022-09-27T16:16:26.4997205Z 
2022-09-27T16:16:26.4997830Z Generating XML reports...
2022-09-27T16:16:26.5028745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161624.xml
2022-09-27T16:16:27.8101912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:27.8102302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:27.8109980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:27.8110501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:27.9603872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapadvjpq
2022-09-27T16:16:27.9605006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapadvjpq/_remote_module_non_scriptable.py
2022-09-27T16:16:28.1816831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:28.1833618Z 
2022-09-27T16:16:28.1834135Z Running tests...
2022-09-27T16:16:28.1834687Z ----------------------------------------------------------------------
2022-09-27T16:16:28.1991323Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10749
2022-09-27T16:16:28.2019650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10750
2022-09-27T16:16:29.2377841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:29.2378255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:29.2545136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:29.2545511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:29.2703549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:29.2704122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:29.2878964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:29.2879346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:29.3925605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtl8de_t
2022-09-27T16:16:29.3927681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtl8de_t/_remote_module_non_scriptable.py
2022-09-27T16:16:29.4133813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx9phyvh
2022-09-27T16:16:29.4135958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx9phyvh/_remote_module_non_scriptable.py
2022-09-27T16:16:29.6174906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:29.6434819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:30.0058403Z skip: Need at least 2 CUDA devices (1.822s)
2022-09-27T16:16:30.0058963Z 
2022-09-27T16:16:30.0059407Z ----------------------------------------------------------------------
2022-09-27T16:16:30.0059659Z Ran 1 test in 1.822s
2022-09-27T16:16:30.0059776Z 
2022-09-27T16:16:30.0059837Z OK (skipped=1)
2022-09-27T16:16:30.0059948Z 
2022-09-27T16:16:30.0060036Z Generating XML reports...
2022-09-27T16:16:30.0092559Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161628.xml
2022-09-27T16:16:31.3074154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:31.3074609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:31.3081936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:31.3082301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:31.4619447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eqmzjl2
2022-09-27T16:16:31.4620539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eqmzjl2/_remote_module_non_scriptable.py
2022-09-27T16:16:31.6834638Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:31.6850569Z 
2022-09-27T16:16:31.6850708Z Running tests...
2022-09-27T16:16:31.6851440Z ----------------------------------------------------------------------
2022-09-27T16:16:31.7021787Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10782
2022-09-27T16:16:31.7049408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10783
2022-09-27T16:16:32.7584977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:32.7585374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:32.7599783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:32.7600128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:32.7918461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:32.7918830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:32.7940503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:32.7940868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:32.9150956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltb8_gk3
2022-09-27T16:16:32.9151416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltb8_gk3/_remote_module_non_scriptable.py
2022-09-27T16:16:32.9194321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdw4cd4pj
2022-09-27T16:16:32.9196142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdw4cd4pj/_remote_module_non_scriptable.py
2022-09-27T16:16:33.1432131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:33.1507444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:33.5087414Z skip: Need at least 2 CUDA devices (1.823s)
2022-09-27T16:16:33.5087655Z 
2022-09-27T16:16:33.5088169Z ----------------------------------------------------------------------
2022-09-27T16:16:33.5088555Z Ran 1 test in 1.824s
2022-09-27T16:16:33.5088671Z 
2022-09-27T16:16:33.5088732Z OK (skipped=1)
2022-09-27T16:16:33.5088842Z 
2022-09-27T16:16:33.5089131Z Generating XML reports...
2022-09-27T16:16:33.5122658Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161631.xml
2022-09-27T16:16:34.8307034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:34.8307663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:34.8316284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:34.8316944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:34.9834844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudr7jlr0
2022-09-27T16:16:34.9835887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudr7jlr0/_remote_module_non_scriptable.py
2022-09-27T16:16:35.2044231Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:35.2060367Z 
2022-09-27T16:16:35.2060956Z Running tests...
2022-09-27T16:16:35.2061594Z ----------------------------------------------------------------------
2022-09-27T16:16:35.2218873Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10815
2022-09-27T16:16:35.2246794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10816
2022-09-27T16:16:36.2394882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:36.2395574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:36.2507558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:36.2508073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:36.2726564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:36.2727196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:36.2844159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:36.2844815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:36.3952040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo0wsy84
2022-09-27T16:16:36.3952992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo0wsy84/_remote_module_non_scriptable.py
2022-09-27T16:16:36.4065064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8pur53z
2022-09-27T16:16:36.4066606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8pur53z/_remote_module_non_scriptable.py
2022-09-27T16:16:36.6224538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:36.6352673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:36.9285194Z skip: Need at least 2 CUDA devices (1.722s)
2022-09-27T16:16:36.9285491Z 
2022-09-27T16:16:36.9285973Z ----------------------------------------------------------------------
2022-09-27T16:16:36.9286413Z Ran 1 test in 1.722s
2022-09-27T16:16:36.9286628Z 
2022-09-27T16:16:36.9286758Z OK (skipped=1)
2022-09-27T16:16:36.9286944Z 
2022-09-27T16:16:36.9287076Z Generating XML reports...
2022-09-27T16:16:36.9317755Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161635.xml
2022-09-27T16:16:38.2448328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:38.2448705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:38.2456655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:38.2457074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:38.3982766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkeu_jj8
2022-09-27T16:16:38.3984328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkeu_jj8/_remote_module_non_scriptable.py
2022-09-27T16:16:38.6199381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:16:38.6215001Z 
2022-09-27T16:16:38.6215129Z Running tests...
2022-09-27T16:16:38.6215827Z ----------------------------------------------------------------------
2022-09-27T16:16:38.6385569Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10848
2022-09-27T16:16:38.6414063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10849
2022-09-27T16:16:39.6666443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:39.6666839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:39.6978370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:39.6978810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:39.7127845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:39.7128336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:39.7427688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:39.7428178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:39.8195514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e42lqze
2022-09-27T16:16:39.8196672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e42lqze/_remote_module_non_scriptable.py
2022-09-27T16:16:39.8640795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g_2xt9l
2022-09-27T16:16:39.8641778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g_2xt9l/_remote_module_non_scriptable.py
2022-09-27T16:16:40.0455428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:40.0923126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:40.4453391Z skip: Need at least 2 CUDA devices (1.824s)
2022-09-27T16:16:40.4453679Z 
2022-09-27T16:16:40.4454164Z ----------------------------------------------------------------------
2022-09-27T16:16:40.4454611Z Ran 1 test in 1.824s
2022-09-27T16:16:40.4454821Z 
2022-09-27T16:16:40.4454976Z OK (skipped=1)
2022-09-27T16:16:40.4455173Z 
2022-09-27T16:16:40.4455316Z Generating XML reports...
2022-09-27T16:16:40.4488846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161638.xml
2022-09-27T16:16:41.0022550Z Running distributed/fsdp/test_fsdp_comm ... [2022-09-27 16:16:41.001895]
2022-09-27T16:16:41.0023108Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:16:41.001982]
2022-09-27T16:16:42.2051088Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-09-27T16:16:42.2062295Z 
2022-09-27T16:16:42.2062901Z Running tests...
2022-09-27T16:16:42.2063544Z ----------------------------------------------------------------------
2022-09-27T16:16:42.2082449Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-09-27T16:16:42.4423134Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10881
2022-09-27T16:16:42.4449029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10882
2022-09-27T16:16:42.4476093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10883
2022-09-27T16:16:42.4505619Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10884
2022-09-27T16:16:43.6345423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:43.6346059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:43.6348264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:43.6348855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:43.6720554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:43.6721197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:43.6724101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:43.6724723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:43.7220462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:43.7221122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:43.7223679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:43.7224494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:43.7487166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:43.7487791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:43.7491212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:43.7491864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:43.7618400Z dist init r=0, world=4
2022-09-27T16:16:43.7981771Z dist init r=3, world=4
2022-09-27T16:16:43.8471892Z dist init r=1, world=4
2022-09-27T16:16:43.8756612Z dist init r=2, world=4
2022-09-27T16:16:43.8893982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:43.8983445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:43.9085962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:43.9087093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:43.9087708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:43.9088223Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:43.9088734Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:43.9097360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:43.9193444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:43.9194107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:43.9194856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:43.9195208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:44.2580070Z skip: Need at least 2 CUDA devices (2.052s)
2022-09-27T16:16:44.2598899Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-09-27T16:16:44.2642892Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10937
2022-09-27T16:16:44.2676149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10938
2022-09-27T16:16:44.2704141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10939
2022-09-27T16:16:44.2731582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10940
2022-09-27T16:16:45.4429923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.4430587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.4434172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.4434805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:45.4463054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.4463746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.4467535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.4468223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:45.4585996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.4586661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.4590282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.4590915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:45.4696998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.4697625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.4701159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.4701768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:45.5693398Z dist init r=2, world=4
2022-09-27T16:16:45.5724649Z dist init r=3, world=4
2022-09-27T16:16:45.5878217Z dist init r=1, world=4
2022-09-27T16:16:45.5942263Z dist init r=0, world=4
2022-09-27T16:16:45.6133544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:45.6234588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:45.6337819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:45.6339174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:45.6339766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:45.6340291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:45.6340810Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:45.6341553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:45.6344154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:45.6344759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:45.6348102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:45.6349404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:45.9772631Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:16:45.9791347Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-09-27T16:16:45.9834724Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10993
2022-09-27T16:16:45.9867049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10994
2022-09-27T16:16:45.9894205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10995
2022-09-27T16:16:45.9922127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10996
2022-09-27T16:16:47.1538488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:47.1539113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:47.1542127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:47.1542803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:47.1553487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:47.1554109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:47.1557718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:47.1558289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:47.1559203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:47.1559702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:47.1563404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:47.1563920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:47.1987815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:47.1988450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:47.1991281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:47.1991841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:47.2805618Z dist init r=3, world=4
2022-09-27T16:16:47.2815154Z dist init r=1, world=4
2022-09-27T16:16:47.2828648Z dist init r=0, world=4
2022-09-27T16:16:47.3448262Z dist init r=2, world=4
2022-09-27T16:16:47.3729466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:47.3830062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:47.3932229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:47.3932911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:47.3934341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:47.3935406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:47.3936352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:47.3937196Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:47.3938692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:47.3939811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:47.3940450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:47.3940975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:47.6963220Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:16:47.6981443Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-09-27T16:16:47.7025557Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11049
2022-09-27T16:16:47.7058019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11050
2022-09-27T16:16:47.7085683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11051
2022-09-27T16:16:47.7113510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11052
2022-09-27T16:16:48.8614186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:48.8616924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:48.8617764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:48.8618371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:48.8724718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:48.8725331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:48.8728883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:48.8729512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:48.8772798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:48.8773374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:48.8777462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:48.8778004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:48.9246069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:48.9246754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:48.9250595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:48.9251281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:48.9874047Z dist init r=0, world=4
2022-09-27T16:16:48.9986192Z dist init r=3, world=4
2022-09-27T16:16:49.0030344Z dist init r=2, world=4
2022-09-27T16:16:49.0523739Z dist init r=1, world=4
2022-09-27T16:16:49.0741567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:49.0833788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:49.0936040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:49.0936839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:49.0937385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:49.0937879Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:49.0938406Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:49.0945036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:49.1042882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:49.1043321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:49.1043702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:49.1044097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:49.4154323Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:16:49.4172467Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-09-27T16:16:49.4215163Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11105
2022-09-27T16:16:49.4247540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11106
2022-09-27T16:16:49.4274931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11107
2022-09-27T16:16:49.4310087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11108
2022-09-27T16:16:50.5670228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:50.5670774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:50.5673586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:50.5674117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:50.6027815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:50.6028489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:50.6032408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:50.6033148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:50.6240116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:50.6240734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:50.6243668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:50.6358126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:50.6359050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:50.6359566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:50.6363622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:50.6364201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:50.6931960Z dist init r=2, world=4
2022-09-27T16:16:50.7286406Z dist init r=1, world=4
2022-09-27T16:16:50.7497771Z dist init r=3, world=4
2022-09-27T16:16:50.7645382Z dist init r=0, world=4
2022-09-27T16:16:50.7843899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:50.8047104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:50.8147586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:50.8148266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:50.8149345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:50.8150291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:50.8151266Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:50.8152062Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:50.8156856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:50.8157503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:50.8158000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:50.8158352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:51.1351850Z skip: Need at least 2 CUDA devices (1.720s)
2022-09-27T16:16:51.1370750Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-09-27T16:16:51.1416359Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11161
2022-09-27T16:16:51.1450680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11162
2022-09-27T16:16:51.1479732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11163
2022-09-27T16:16:51.1508355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11164
2022-09-27T16:16:52.3186237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:52.3186946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:52.3189728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:52.3190283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.3474700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:52.3475371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:52.3478352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:52.3479018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.3533907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:52.3534553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:52.3538197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:52.3538833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.3693215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:52.3694028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:52.3697663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:52.3698509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.4452746Z dist init r=0, world=4
2022-09-27T16:16:52.4728865Z dist init r=2, world=4
2022-09-27T16:16:52.4789841Z dist init r=1, world=4
2022-09-27T16:16:52.4981077Z dist init r=3, world=4
2022-09-27T16:16:52.5105800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:52.5106232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:52.5106606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:52.5106980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:52.5107610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:52.5108159Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:52.5108686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:52.5109193Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:52.5208134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:52.5208527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:52.5208883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:52.5209230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:52.8550219Z skip: Need at least 2 CUDA devices (1.720s)
2022-09-27T16:16:52.8569248Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-09-27T16:16:52.8614003Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11217
2022-09-27T16:16:52.8646876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11218
2022-09-27T16:16:52.8675200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11219
2022-09-27T16:16:52.8704003Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11220
2022-09-27T16:16:54.0055718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:54.0056374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:54.0059595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:54.0060254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:54.0348492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:54.0349185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:54.0352483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:54.0353092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:54.0721598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:54.0722249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:54.0725578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:54.0726172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:54.1031883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:54.1032517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:54.1035438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:54.1036069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:54.1328624Z dist init r=1, world=4
2022-09-27T16:16:54.1644117Z dist init r=0, world=4
2022-09-27T16:16:54.1982717Z dist init r=2, world=4
2022-09-27T16:16:54.2272036Z dist init r=3, world=4
2022-09-27T16:16:54.2379724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:54.2393883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:54.2394794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:54.2395223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:54.2395649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:54.2396142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:54.2396667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:54.2482867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:54.2501825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:54.2502386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:54.2502914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:54.2503452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:54.5745823Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:16:54.5763916Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-09-27T16:16:54.5801357Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11273
2022-09-27T16:16:54.5830490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11274
2022-09-27T16:16:54.5857772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11275
2022-09-27T16:16:54.5885432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11276
2022-09-27T16:16:55.7885822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:55.7886437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:55.7889037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:55.7889640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:55.8647761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:55.8648418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:55.8651325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:55.8651968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:55.8692456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:55.8693214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:55.8696718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:55.8697314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:55.8977642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:55.8978314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:55.8981339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:55.8982009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:55.9149964Z dist init r=0, world=4
2022-09-27T16:16:55.9905545Z dist init r=3, world=4
2022-09-27T16:16:55.9980150Z dist init r=1, world=4
2022-09-27T16:16:56.0214093Z dist init r=2, world=4
2022-09-27T16:16:56.0415289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:56.0515925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:56.0617899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:56.0618589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:56.0619510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:56.0620243Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:56.0620832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:56.0621353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:56.0724870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:56.0725553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:56.0726096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:56.0726610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:56.3927457Z skip: Need at least 2 CUDA devices (1.818s)
2022-09-27T16:16:56.3927710Z 
2022-09-27T16:16:56.3928043Z ----------------------------------------------------------------------
2022-09-27T16:16:56.3928357Z Ran 8 tests in 14.186s
2022-09-27T16:16:56.3928475Z 
2022-09-27T16:16:56.3928548Z OK (skipped=8)
2022-09-27T16:16:56.3928649Z 
2022-09-27T16:16:56.3928734Z Generating XML reports...
2022-09-27T16:16:56.3971761Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220927161642.xml
2022-09-27T16:16:56.6851279Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-09-27 16:16:56.684713]
2022-09-27T16:16:56.6852157Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:16:56.684804]
2022-09-27T16:16:57.8856748Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-09-27T16:16:57.8869635Z 
2022-09-27T16:16:57.8869778Z Running tests...
2022-09-27T16:16:57.8870425Z ----------------------------------------------------------------------
2022-09-27T16:16:57.8880680Z   test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-09-27T16:16:57.8887317Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-09-27T16:16:57.8907756Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s)
2022-09-27T16:16:58.1234546Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11340
2022-09-27T16:16:58.1260220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11341
2022-09-27T16:16:58.1287335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11342
2022-09-27T16:16:58.1316601Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11343
2022-09-27T16:16:59.3348114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:59.3348756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:59.3352469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:59.3353217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:59.3582590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:59.3583262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:59.3587301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:59.3587933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:59.3749500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:59.3750111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:59.3754452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:59.3755095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:59.4356280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:59.4356925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:59.4359249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:59.4359896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:59.4614533Z dist init r=0, world=4
2022-09-27T16:16:59.4884996Z dist init r=1, world=4
2022-09-27T16:16:59.5010736Z dist init r=3, world=4
2022-09-27T16:16:59.5597239Z dist init r=2, world=4
2022-09-27T16:16:59.5806802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:59.5907812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:16:59.5908729Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:59.5909253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:59.5909720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:16:59.5910507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:59.5911227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:59.5911739Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:16:59.6015919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:16:59.6016465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:16:59.6016953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:59.6017709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:59.9362983Z skip: Need at least 2 CUDA devices (2.045s)
2022-09-27T16:16:59.9416785Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11396
2022-09-27T16:16:59.9450094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11397
2022-09-27T16:16:59.9477199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11398
2022-09-27T16:16:59.9505042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11399
2022-09-27T16:17:01.0889196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:01.0889864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:01.0893386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:01.0894001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:01.1015844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:01.1016555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:01.1021164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:01.1021761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:01.1632370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:01.1632998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:01.1635915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:01.1636542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:01.1715433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:01.1716076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:01.1720535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:01.1721162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:01.2162766Z dist init r=3, world=4
2022-09-27T16:17:01.2320203Z dist init r=1, world=4
2022-09-27T16:17:01.2886674Z dist init r=0, world=4
2022-09-27T16:17:01.2973751Z dist init r=2, world=4
2022-09-27T16:17:01.3175116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:01.3377868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:01.3379339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:01.3379848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:01.3380538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:01.3382641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:01.3384206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:01.3385092Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:01.3387306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:01.3387941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:01.3388401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:01.3388921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:01.6547329Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:01.6600318Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11452
2022-09-27T16:17:01.6633749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11453
2022-09-27T16:17:01.6661998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11454
2022-09-27T16:17:01.6690661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11455
2022-09-27T16:17:02.8274948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.8275796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.8278012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.8278610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.8295361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.8296001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.8300682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.8301309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.8314334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.8314933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.8318579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.8319191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.8437108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.8437736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.8441910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.8442559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.9533600Z dist init r=0, world=4
2022-09-27T16:17:02.9560441Z dist init r=1, world=4
2022-09-27T16:17:02.9599986Z dist init r=3, world=4
2022-09-27T16:17:02.9674186Z dist init r=2, world=4
2022-09-27T16:17:02.9809276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:02.9971067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:03.0073140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:03.0073667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:03.0074381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:03.0074907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:03.0075433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:03.0113623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:03.0180204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:03.0180758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:03.0181321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:03.0182156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:03.3731863Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:03.3784819Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11508
2022-09-27T16:17:03.3817786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11509
2022-09-27T16:17:03.3845176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11510
2022-09-27T16:17:03.3873073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11511
2022-09-27T16:17:04.5250099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:04.5250711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:04.5253929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:04.5254484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:04.5559627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:04.5560255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:04.5563020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:04.5563582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:04.5672590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:04.5673281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:04.5677940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:04.5678618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:04.5928504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:04.5933010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:04.5933846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:04.5934430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:04.6514639Z dist init r=0, world=4
2022-09-27T16:17:04.6818546Z dist init r=3, world=4
2022-09-27T16:17:04.6953487Z dist init r=2, world=4
2022-09-27T16:17:04.7167991Z dist init r=1, world=4
2022-09-27T16:17:04.7329083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:04.7529193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:04.7530084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:04.7530438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:04.7531056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:04.7531591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:04.7532325Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:04.7533052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:04.7636117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:04.7636792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:04.7637322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:04.7637841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:05.0914920Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:05.0968365Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11564
2022-09-27T16:17:05.1003040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11565
2022-09-27T16:17:05.1030754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11566
2022-09-27T16:17:05.1058550Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11567
2022-09-27T16:17:06.2693885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:06.2694417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:06.2697643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:06.2698191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:06.2698875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:06.2699346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:06.2703701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:06.2704481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:06.2964380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:06.2965026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:06.2969588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:06.2970243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:06.3011431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:06.3012035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:06.3016469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:06.3017113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:06.3954243Z dist init r=1, world=4
2022-09-27T16:17:06.3962235Z dist init r=2, world=4
2022-09-27T16:17:06.4220511Z dist init r=0, world=4
2022-09-27T16:17:06.4293559Z dist init r=3, world=4
2022-09-27T16:17:06.4402165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:06.4565314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:06.4667479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:06.4668052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:06.4668728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:06.4669272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:06.4669786Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:06.4706860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:06.4777352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:06.4777760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:06.4778103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:06.4778450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:06.8101001Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:06.8153958Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11620
2022-09-27T16:17:06.8187830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11621
2022-09-27T16:17:06.8216778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11622
2022-09-27T16:17:06.8245279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11623
2022-09-27T16:17:07.9811544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:07.9812225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:07.9815564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:07.9816182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:07.9842023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:07.9842667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:07.9847859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:07.9848486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:07.9988952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:07.9989618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:07.9994566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:07.9995201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:08.0021538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:08.0022161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:08.0026608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:08.0027401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:08.1147251Z dist init r=1, world=4
2022-09-27T16:17:08.1238829Z dist init r=0, world=4
2022-09-27T16:17:08.1315466Z dist init r=2, world=4
2022-09-27T16:17:08.1319405Z dist init r=3, world=4
2022-09-27T16:17:08.1525306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:08.1626499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:08.1629831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:08.1630359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:08.1630978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:08.1631524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:08.1632044Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:08.1632572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:08.1635194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:08.1635668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:08.1636193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:08.1636733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:08.4285076Z skip: Need at least 2 CUDA devices (1.618s)
2022-09-27T16:17:08.4336060Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11676
2022-09-27T16:17:08.4367647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11677
2022-09-27T16:17:08.4395155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11678
2022-09-27T16:17:08.4422654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11679
2022-09-27T16:17:09.6253085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.6253799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.6256629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.6257275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:09.6405092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.6405779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.6410401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.6410956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:09.6422585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.6423121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.6428294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.6428813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:09.6757032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.6757702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.6761622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.6762250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:09.7513314Z dist init r=3, world=4
2022-09-27T16:17:09.7669672Z dist init r=1, world=4
2022-09-27T16:17:09.7712068Z dist init r=2, world=4
2022-09-27T16:17:09.8003161Z dist init r=0, world=4
2022-09-27T16:17:09.8281624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:09.8326752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:09.8328067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:09.8328718Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:09.8329121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:09.8329616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:09.8330126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:09.8383301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:09.8435104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:09.8435819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:09.8436373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:09.8436897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:10.1464056Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:10.1507609Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11732
2022-09-27T16:17:10.1535013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11733
2022-09-27T16:17:10.1561577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11734
2022-09-27T16:17:10.1588391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11735
2022-09-27T16:17:11.3461307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:11.3461939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:11.3465211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:11.3465845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:11.3504953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:11.3505550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:11.3509627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:11.3510513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:11.3601989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:11.3602871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:11.3607607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:11.3608206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:11.3660406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:11.3660991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:11.3665419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:11.3666051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:11.4715472Z dist init r=1, world=4
2022-09-27T16:17:11.4753171Z dist init r=3, world=4
2022-09-27T16:17:11.4889544Z dist init r=2, world=4
2022-09-27T16:17:11.4909917Z dist init r=0, world=4
2022-09-27T16:17:11.5097847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:11.5301101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:11.5301744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:11.5302417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:11.5305168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:11.5306035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:11.5307058Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:11.5308124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:11.5308851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:11.5309703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:11.5310277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:11.5310818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:11.8629861Z skip: Need at least 2 CUDA devices (1.716s)
2022-09-27T16:17:11.8630138Z 
2022-09-27T16:17:11.8630652Z ----------------------------------------------------------------------
2022-09-27T16:17:11.8631132Z Ran 11 tests in 13.976s
2022-09-27T16:17:11.8631351Z 
2022-09-27T16:17:11.8631452Z OK (skipped=11)
2022-09-27T16:17:11.8631551Z 
2022-09-27T16:17:11.8631639Z Generating XML reports...
2022-09-27T16:17:11.8668216Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220927161657.xml
2022-09-27T16:17:11.8677661Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220927161657.xml
2022-09-27T16:17:12.1542444Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-09-27 16:17:12.153810]
2022-09-27T16:17:12.1543286Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:12.153887]
2022-09-27T16:17:13.3544491Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-09-27T16:17:13.3555658Z 
2022-09-27T16:17:13.3556275Z Running tests...
2022-09-27T16:17:13.3556684Z ----------------------------------------------------------------------
2022-09-27T16:17:13.5898698Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11799
2022-09-27T16:17:13.5925321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11800
2022-09-27T16:17:13.5952121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11801
2022-09-27T16:17:13.5980762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11802
2022-09-27T16:17:14.7765442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:14.7766104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:14.7767624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:14.7768273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:14.8031424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:14.8032082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:14.8036272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:14.8036921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:14.8044940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:14.8045543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:14.8049206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:14.8049803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:14.8058462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:14.8059045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:14.8064120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:14.8064780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:14.9024916Z dist init r=0, world=4
2022-09-27T16:17:14.9290581Z dist init r=2, world=4
2022-09-27T16:17:14.9320022Z dist init r=1, world=4
2022-09-27T16:17:14.9328238Z dist init r=3, world=4
2022-09-27T16:17:14.9636550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:14.9737498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:14.9840193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:14.9840866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:14.9841685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:14.9842304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:14.9842988Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:14.9843494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:14.9847405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:14.9848185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:14.9848696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:14.9849205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:15.3025290Z skip: Need at least 2 CUDA devices (1.947s)
2022-09-27T16:17:15.3079876Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11855
2022-09-27T16:17:15.3108628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11856
2022-09-27T16:17:15.3136699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11857
2022-09-27T16:17:15.3172542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11858
2022-09-27T16:17:16.4481155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.4481775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.4485007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.4485628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.4904277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.4904972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.4907679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.4908332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.5025573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.5026165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.5030163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.5030749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.5677278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.5677942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.5681430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.5682046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.5750783Z dist init r=2, world=4
2022-09-27T16:17:16.6166307Z dist init r=0, world=4
2022-09-27T16:17:16.6275784Z dist init r=3, world=4
2022-09-27T16:17:16.6948012Z dist init r=1, world=4
2022-09-27T16:17:16.7087795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:16.7188669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:16.7290796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:16.7291497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:16.7292513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:16.7293249Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:16.7293927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:16.7294454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:16.7398516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:16.7399543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:16.7401952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:16.7402869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:17.0214116Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:17:17.0274906Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11911
2022-09-27T16:17:17.0307732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11912
2022-09-27T16:17:17.0335216Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11913
2022-09-27T16:17:17.0364345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11914
2022-09-27T16:17:18.2403025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:18.2403652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:18.2406215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:18.2406964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:18.2418754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:18.2419327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:18.2424483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:18.2425091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:18.2576861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:18.2577489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:18.2581427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:18.2582056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:18.2695713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:18.2696310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:18.2700604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:18.2701208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:18.3663415Z dist init r=2, world=4
2022-09-27T16:17:18.3717989Z dist init r=0, world=4
2022-09-27T16:17:18.3831182Z dist init r=3, world=4
2022-09-27T16:17:18.4061425Z dist init r=1, world=4
2022-09-27T16:17:18.4242017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:18.4341946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:18.4373010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:18.4373713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:18.4374627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:18.4375164Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:18.4445332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:18.4446056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:18.4550375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:18.4551077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:18.4551443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:18.4551789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:18.7405905Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:17:18.7467094Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11967
2022-09-27T16:17:18.7498693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11968
2022-09-27T16:17:18.7526323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11969
2022-09-27T16:17:18.7554306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11970
2022-09-27T16:17:19.9016795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:19.9017439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:19.9020845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:19.9021528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:19.9028043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:19.9028634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:19.9032798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:19.9033428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:19.9201220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:19.9201825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:19.9205788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:19.9206407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:19.9214451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:19.9214998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:19.9220281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:19.9220999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:20.0282670Z dist init r=3, world=4
2022-09-27T16:17:20.0291486Z dist init r=0, world=4
2022-09-27T16:17:20.0459239Z dist init r=1, world=4
2022-09-27T16:17:20.0485353Z dist init r=2, world=4
2022-09-27T16:17:20.0690917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:20.0796195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:20.0898417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:20.0899065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:20.0899751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:20.0900267Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:20.0900794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:20.0995520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:20.1005211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:20.1006019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:20.1007027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:20.1007545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:20.3594031Z skip: Need at least 2 CUDA devices (1.619s)
2022-09-27T16:17:20.3649594Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12023
2022-09-27T16:17:20.3682152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12024
2022-09-27T16:17:20.3709333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12025
2022-09-27T16:17:20.3737363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12026
2022-09-27T16:17:21.5098724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:21.5099408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:21.5103217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:21.5103999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:21.5609509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:21.5610162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:21.5613034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:21.5613664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:21.5729510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:21.5730212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:21.5734353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:21.5734983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:21.5923296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:21.5923914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:21.5928247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:21.5928864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:21.6400047Z dist init r=3, world=4
2022-09-27T16:17:21.6865061Z dist init r=2, world=4
2022-09-27T16:17:21.6984563Z dist init r=0, world=4
2022-09-27T16:17:21.7193103Z dist init r=1, world=4
2022-09-27T16:17:21.7476506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:21.7579231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:21.7580358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:21.7581253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:21.7581895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:21.7582792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:21.7583600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:21.7584452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:21.7588825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:21.7590942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:21.7591594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:21.7592246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:22.0777789Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:22.0836923Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12079
2022-09-27T16:17:22.0869752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12080
2022-09-27T16:17:22.0896862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12081
2022-09-27T16:17:22.0925291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12082
2022-09-27T16:17:23.1903207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:23.1904014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:23.1907024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:23.1907621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:23.2576023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:23.2576679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:23.2579622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:23.2580253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:23.2730703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:23.2731298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:23.2735039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:23.2735661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:23.2843515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:23.2844128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:23.2848061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:23.2848715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:23.3171485Z dist init r=3, world=4
2022-09-27T16:17:23.3834287Z dist init r=0, world=4
2022-09-27T16:17:23.4015739Z dist init r=1, world=4
2022-09-27T16:17:23.4084982Z dist init r=2, world=4
2022-09-27T16:17:23.4284811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:23.4326165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:23.4326788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:23.4327564Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:23.4327984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:23.4328488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:23.4329014Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:23.4386450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:23.4433924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:23.4434607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:23.4435125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:23.4435663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:23.7966028Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:17:23.8025793Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12135
2022-09-27T16:17:23.8058318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12136
2022-09-27T16:17:23.8086380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12137
2022-09-27T16:17:23.8114526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12138
2022-09-27T16:17:24.9353179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:24.9353782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:24.9356988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:24.9357629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:24.9874231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:24.9874834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:24.9878233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:24.9878821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:25.0141173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:25.0141800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:25.0145829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:25.0146469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:25.0308114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:25.0308723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:25.0313121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:25.0313797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:25.0633012Z dist init r=0, world=4
2022-09-27T16:17:25.1132884Z dist init r=1, world=4
2022-09-27T16:17:25.1399477Z dist init r=2, world=4
2022-09-27T16:17:25.1588141Z dist init r=3, world=4
2022-09-27T16:17:25.1748443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:25.1748850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:25.1849735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:25.1850335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:25.1851283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:25.1852048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:25.1852585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:25.1853284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:25.1960690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:25.1961106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:25.1961466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:25.1961869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:25.5155451Z skip: Need at least 2 CUDA devices (1.719s)
2022-09-27T16:17:25.5172856Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-09-27T16:17:25.5173851Z 
2022-09-27T16:17:25.5174085Z ----------------------------------------------------------------------
2022-09-27T16:17:25.5174341Z Ran 8 tests in 12.162s
2022-09-27T16:17:25.5174460Z 
2022-09-27T16:17:25.5174537Z OK (skipped=8)
2022-09-27T16:17:25.5174631Z 
2022-09-27T16:17:25.5174735Z Generating XML reports...
2022-09-27T16:17:25.5216654Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220927161713.xml
2022-09-27T16:17:25.8078156Z Running distributed/fsdp/test_fsdp_meta ... [2022-09-27 16:17:25.807425]
2022-09-27T16:17:25.8078744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:25.807501]
2022-09-27T16:17:27.0081302Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-09-27T16:17:27.0093515Z 
2022-09-27T16:17:27.0093962Z Running tests...
2022-09-27T16:17:27.0094412Z ----------------------------------------------------------------------
2022-09-27T16:17:27.2484706Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12202
2022-09-27T16:17:27.2511680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12203
2022-09-27T16:17:28.3678708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:28.3679352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:28.3681265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:28.3681810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:28.3808067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:28.3808552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:28.3812260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:28.3812847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:28.4909585Z dist init r=0, world=2
2022-09-27T16:17:28.5029086Z dist init r=1, world=2
2022-09-27T16:17:28.5137316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:28.5138070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:28.5138673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:28.5139212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:28.5242952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:28.5243588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:28.7548369Z skip: Need at least 2 CUDA devices (1.745s)
2022-09-27T16:17:28.7554820Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-09-27T16:17:28.7603233Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12230
2022-09-27T16:17:28.7635515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12231
2022-09-27T16:17:29.8574430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:29.8575026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:29.8577548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:29.8578109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:29.8669774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:29.8670387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:29.8674028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:29.8674649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:29.9804823Z dist init r=0, world=2
2022-09-27T16:17:29.9891280Z dist init r=1, world=2
2022-09-27T16:17:29.9999620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:30.0000117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:30.0000792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:30.0001506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:30.0105925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:30.0106682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:30.2666510Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:17:30.2719899Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12258
2022-09-27T16:17:30.2752339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12259
2022-09-27T16:17:31.3874782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:31.3875450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:31.3878558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:31.3879176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:31.4008333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:31.4008946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:31.4012226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:31.4012880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:31.5114173Z dist init r=0, world=2
2022-09-27T16:17:31.5272302Z dist init r=1, world=2
2022-09-27T16:17:31.5424148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:31.5424894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:31.5425811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:31.5426650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:31.5429766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:31.5430167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:31.7786636Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:17:31.7835995Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12286
2022-09-27T16:17:31.7867653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12287
2022-09-27T16:17:32.8690058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:32.8690460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:32.8693117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:32.8693663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:32.8696976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:32.8697487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:32.8701915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:32.8702489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:32.9934458Z dist init r=0, world=2
2022-09-27T16:17:32.9942061Z dist init r=1, world=2
2022-09-27T16:17:33.0144247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:33.0144785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:33.0145673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:33.0146519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:33.0151227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:33.0151806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:33.2899625Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:17:33.2950439Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12314
2022-09-27T16:17:33.2982823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12315
2022-09-27T16:17:34.3829551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:34.3832030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:34.3832865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:34.3833472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:34.3952220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:34.3952847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:34.3956837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:34.3957312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:34.5057805Z dist init r=0, world=2
2022-09-27T16:17:34.5184597Z dist init r=1, world=2
2022-09-27T16:17:34.5367458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:34.5367869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:34.5368589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:34.5369110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:34.5472896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:34.5473467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:34.8014266Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:17:34.8021427Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-09-27T16:17:34.8026439Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-09-27T16:17:34.8031224Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-09-27T16:17:34.8035813Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-09-27T16:17:34.8082802Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12342
2022-09-27T16:17:34.8116116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12343
2022-09-27T16:17:35.8737564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:35.8738233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:35.8740413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:35.8740766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:35.8949334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:35.8949695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:35.8952488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:35.8952920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:35.9994997Z dist init r=1, world=2
2022-09-27T16:17:36.0235964Z dist init r=0, world=2
2022-09-27T16:17:36.0444928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:36.0445824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:36.0446614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:36.0447170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:36.0450683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:36.0451060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:36.3151476Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:17:36.3201757Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12370
2022-09-27T16:17:36.3234194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12371
2022-09-27T16:17:37.3859994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:37.3860420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:37.3863140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:37.3863623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:37.4334461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:37.4334878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:37.4337819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:37.4338252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:37.5102161Z dist init r=1, world=2
2022-09-27T16:17:37.5602008Z dist init r=0, world=2
2022-09-27T16:17:37.5811573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:37.5812016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:37.5812609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:37.5813238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:37.5817773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:37.5818244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:37.8268642Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:17:37.8274686Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-09-27T16:17:37.8278867Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-09-27T16:17:37.8279422Z 
2022-09-27T16:17:37.8279875Z ----------------------------------------------------------------------
2022-09-27T16:17:37.8280318Z Ran 14 tests in 10.819s
2022-09-27T16:17:37.8280530Z 
2022-09-27T16:17:37.8280665Z OK (skipped=14)
2022-09-27T16:17:37.8280805Z 
2022-09-27T16:17:37.8280894Z Generating XML reports...
2022-09-27T16:17:37.8326026Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220927161727.xml
2022-09-27T16:17:38.1234064Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-09-27 16:17:38.122966]
2022-09-27T16:17:38.1234697Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:38.123046]
2022-09-27T16:17:39.3192231Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-09-27T16:17:39.3203768Z 
2022-09-27T16:17:39.3204252Z Running tests...
2022-09-27T16:17:39.3204819Z ----------------------------------------------------------------------
2022-09-27T16:17:39.3214168Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules)
2022-09-27T16:17:39.5550131Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12409
2022-09-27T16:17:39.5576610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12410
2022-09-27T16:17:39.5603382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12411
2022-09-27T16:17:39.5632695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12412
2022-09-27T16:17:40.7206999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.7207622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.7210784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.7211323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.7511822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.7512474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.7515828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.7516467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.7778926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.7779558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.7783510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.7784398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.8477264Z dist init r=1, world=4
2022-09-27T16:17:40.8518281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.8519117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.8522022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.8522856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.8772335Z dist init r=2, world=4
2022-09-27T16:17:40.9037595Z dist init r=0, world=4
2022-09-27T16:17:40.9790535Z dist init r=3, world=4
2022-09-27T16:17:40.9951532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:40.9951926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:41.0054068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:41.0054682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:41.0055634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:41.0056329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:41.0057021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:41.0057795Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:41.0061339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:41.0061969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:41.0062345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:41.0062869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:41.3683002Z skip: Need at least 2 CUDA devices (2.048s)
2022-09-27T16:17:41.3692835Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules)
2022-09-27T16:17:41.3736478Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12465
2022-09-27T16:17:41.3770474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12466
2022-09-27T16:17:41.3798032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12467
2022-09-27T16:17:41.3825926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12468
2022-09-27T16:17:42.5491541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:42.5492271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:42.5495567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:42.5496211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:42.5630886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:42.5631521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:42.5636088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:42.5636728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:42.5971240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:42.5971894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:42.5976234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:42.5976882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:42.6004966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:42.6005756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:42.6010793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:42.6011443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:42.6756685Z dist init r=3, world=4
2022-09-27T16:17:42.6906732Z dist init r=2, world=4
2022-09-27T16:17:42.7223798Z dist init r=1, world=4
2022-09-27T16:17:42.7283750Z dist init r=0, world=4
2022-09-27T16:17:42.7634824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:42.7735745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:42.7837998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:42.7839208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:42.7840268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:42.7840995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:42.7841721Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:42.7842323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:42.7846731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:42.7847308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:42.7847864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:42.7848237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:43.0869298Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:43.0875743Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-09-27T16:17:43.0922590Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12521
2022-09-27T16:17:43.0955961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12522
2022-09-27T16:17:43.0983144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12523
2022-09-27T16:17:43.1011633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12524
2022-09-27T16:17:44.2885315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:44.2885970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:44.2889605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:44.2890238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:44.2934769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:44.2935368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:44.2939637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:44.2940275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:44.2965967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:44.2966582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:44.2970640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:44.2971273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:44.3298892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:44.3299498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:44.3303112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:44.3303742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:44.4146647Z dist init r=2, world=4
2022-09-27T16:17:44.4185231Z dist init r=0, world=4
2022-09-27T16:17:44.4214192Z dist init r=1, world=4
2022-09-27T16:17:44.4576536Z dist init r=3, world=4
2022-09-27T16:17:44.4725643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:44.4726021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:44.4827680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:44.4828379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:44.4829296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:44.4829932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:44.4830767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:44.4831291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:44.4935225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:44.4935779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:44.4936333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:44.4936867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:44.8052379Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:44.8061920Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-09-27T16:17:44.8105812Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12577
2022-09-27T16:17:44.8136980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12578
2022-09-27T16:17:44.8164369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12579
2022-09-27T16:17:44.8192556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12580
2022-09-27T16:17:45.9380019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:45.9380672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:45.9384406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:45.9385059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:45.9870972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:45.9871919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:45.9874276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:45.9875101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:45.9911054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:45.9911655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:45.9915592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:45.9916173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:46.0024010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:46.0024693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:46.0028695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:46.0029327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:46.0654084Z dist init r=2, world=4
2022-09-27T16:17:46.1163766Z dist init r=3, world=4
2022-09-27T16:17:46.1164002Z dist init r=0, world=4
2022-09-27T16:17:46.1268819Z dist init r=1, world=4
2022-09-27T16:17:46.1579467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:46.1680518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:46.1781992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:46.1782881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:46.1783450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:46.1784223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:46.1784930Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:46.1785453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:46.1792607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:46.1793116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:46.1793867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:46.1794445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:46.5234211Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:46.5245022Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-09-27T16:17:46.5290462Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12633
2022-09-27T16:17:46.5324022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12634
2022-09-27T16:17:46.5351022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12635
2022-09-27T16:17:46.5378790Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12636
2022-09-27T16:17:47.6903752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.6904647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.6908615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.6909268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:47.7292718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.7293366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.7296438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.7297035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:47.7298864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.7299436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.7304079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.7304783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:47.7545224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.7545884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.7549611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.7550283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:47.8166852Z dist init r=1, world=4
2022-09-27T16:17:47.8558417Z dist init r=3, world=4
2022-09-27T16:17:47.8558748Z dist init r=2, world=4
2022-09-27T16:17:47.8817653Z dist init r=0, world=4
2022-09-27T16:17:47.9078837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:47.9179906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:47.9282294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:17:47.9282996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:47.9283931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:47.9284471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:47.9285009Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:47.9285535Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:17:47.9291949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:17:47.9292513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:47.9293067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:47.9293644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:48.2420529Z skip: Need at least 2 CUDA devices (1.718s)
2022-09-27T16:17:48.2420871Z 
2022-09-27T16:17:48.2421384Z ----------------------------------------------------------------------
2022-09-27T16:17:48.2421663Z Ran 5 tests in 8.922s
2022-09-27T16:17:48.2421780Z 
2022-09-27T16:17:48.2421857Z OK (skipped=5)
2022-09-27T16:17:48.2421953Z 
2022-09-27T16:17:48.2422040Z Generating XML reports...
2022-09-27T16:17:48.2459485Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220927161739.xml
2022-09-27T16:17:48.5342822Z Running distributed/algorithms/quantization/test_quantization ... [2022-09-27 16:17:48.533874]
2022-09-27T16:17:48.5388381Z MPI not available -- MPI backend tests will be skipped
2022-09-27T16:17:48.5439975Z Running distributed tests for the test backend with env init_method
2022-09-27T16:17:48.5455022Z Running distributed tests for the gloo backend with env init_method
2022-09-27T16:17:48.5461626Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:48.545889]
2022-09-27T16:17:48.5474541Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:17:48.547149]
2022-09-27T16:17:49.7694466Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:17:49.7695257Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7695606Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7695860Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7696160Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7696457Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7696740Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:17:49.7697033Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:17:49.7708752Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:17:50.7785627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:50.7786209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:50.7788071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:50.7788661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:50.9312439Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:17:50.9321961Z 
2022-09-27T16:17:50.9322261Z Running tests...
2022-09-27T16:17:50.9322904Z ----------------------------------------------------------------------
2022-09-27T16:17:51.1703768Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12726
2022-09-27T16:17:51.1729619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12727
2022-09-27T16:17:51.1756166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12728
2022-09-27T16:17:52.3053763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:52.3054284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:52.3055794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:52.3056318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:52.3085061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:52.3085558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:52.3088888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:52.3089670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:52.3623681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:52.3624869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:52.3626073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:52.3626721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:52.4288425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:52.4306574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:52.4832403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:52.5119302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:52.5119940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:52.5120886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:52.5121590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:52.5122365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:52.5123269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:52.8796564Z ok (1.947s)
2022-09-27T16:17:52.8796818Z 
2022-09-27T16:17:52.8797355Z ----------------------------------------------------------------------
2022-09-27T16:17:52.8797650Z Ran 1 test in 1.947s
2022-09-27T16:17:52.8797769Z 
2022-09-27T16:17:52.8797832Z OK
2022-09-27T16:17:52.8797926Z 
2022-09-27T16:17:52.8798030Z Generating XML reports...
2022-09-27T16:17:52.8830924Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161750.xml
2022-09-27T16:17:54.1826087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:54.1826719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:54.1828123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:54.1828688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:54.3343604Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:17:54.3353315Z 
2022-09-27T16:17:54.3353766Z Running tests...
2022-09-27T16:17:54.3354344Z ----------------------------------------------------------------------
2022-09-27T16:17:54.5687336Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12779
2022-09-27T16:17:54.5713326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12780
2022-09-27T16:17:54.5739576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12781
2022-09-27T16:17:55.6412891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:55.6413280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:55.6415699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:55.6416170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:55.6513449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:55.6514182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:55.6517164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:55.6517964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:55.6779493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:55.6780139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:55.6780903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:55.6781422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:55.7656279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:55.7757115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:55.7994990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:17:55.8167574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:55.8168147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:55.8168519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:17:55.8169145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:55.8169667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:55.8170193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:17:56.1782015Z ok (1.842s)
2022-09-27T16:17:56.1782198Z 
2022-09-27T16:17:56.1782522Z ----------------------------------------------------------------------
2022-09-27T16:17:56.1782763Z Ran 1 test in 1.843s
2022-09-27T16:17:56.1782891Z 
2022-09-27T16:17:56.1782954Z OK
2022-09-27T16:17:56.1783048Z 
2022-09-27T16:17:56.1783140Z Generating XML reports...
2022-09-27T16:17:56.1816263Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161754.xml
2022-09-27T16:17:57.4712249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:57.4712628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:57.4714535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:57.4715133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:57.6216385Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:17:57.6225579Z 
2022-09-27T16:17:57.6225886Z Running tests...
2022-09-27T16:17:57.6226559Z ----------------------------------------------------------------------
2022-09-27T16:17:57.6234781Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:17:57.6235191Z 
2022-09-27T16:17:57.6235609Z ----------------------------------------------------------------------
2022-09-27T16:17:57.6235970Z Ran 1 test in 0.001s
2022-09-27T16:17:57.6236087Z 
2022-09-27T16:17:57.6236152Z OK (skipped=1)
2022-09-27T16:17:57.6236262Z 
2022-09-27T16:17:57.6236348Z Generating XML reports...
2022-09-27T16:17:57.6259788Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161757.xml
2022-09-27T16:17:58.8083440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:58.8083883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:58.8084970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:58.8085421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:58.9612020Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:17:58.9620795Z 
2022-09-27T16:17:58.9620920Z Running tests...
2022-09-27T16:17:58.9621499Z ----------------------------------------------------------------------
2022-09-27T16:17:58.9630846Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:17:58.9631239Z 
2022-09-27T16:17:58.9631718Z ----------------------------------------------------------------------
2022-09-27T16:17:58.9632085Z Ran 1 test in 0.001s
2022-09-27T16:17:58.9632189Z 
2022-09-27T16:17:58.9632264Z OK (skipped=1)
2022-09-27T16:17:58.9632375Z 
2022-09-27T16:17:58.9632461Z Generating XML reports...
2022-09-27T16:17:58.9655863Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161758.xml
2022-09-27T16:18:00.1542718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:00.1543165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:00.1544900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:00.1545298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:00.3044118Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:00.3052743Z 
2022-09-27T16:18:00.3053112Z Running tests...
2022-09-27T16:18:00.3053520Z ----------------------------------------------------------------------
2022-09-27T16:18:00.3061687Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:00.3062062Z 
2022-09-27T16:18:00.3062337Z ----------------------------------------------------------------------
2022-09-27T16:18:00.3062605Z Ran 1 test in 0.001s
2022-09-27T16:18:00.3062724Z 
2022-09-27T16:18:00.3062804Z OK (skipped=1)
2022-09-27T16:18:00.3062914Z 
2022-09-27T16:18:00.3062986Z Generating XML reports...
2022-09-27T16:18:00.3087023Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161800.xml
2022-09-27T16:18:01.4981586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:01.4982052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:01.4983308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:01.4983750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:01.6488871Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:01.6497779Z 
2022-09-27T16:18:01.6497858Z Running tests...
2022-09-27T16:18:01.6498824Z ----------------------------------------------------------------------
2022-09-27T16:18:01.6507562Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:01.6507916Z 
2022-09-27T16:18:01.6508349Z ----------------------------------------------------------------------
2022-09-27T16:18:01.6508742Z Ran 1 test in 0.001s
2022-09-27T16:18:01.6508858Z 
2022-09-27T16:18:01.6508931Z OK (skipped=1)
2022-09-27T16:18:01.6509038Z 
2022-09-27T16:18:01.6509341Z Generating XML reports...
2022-09-27T16:18:01.6533701Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161801.xml
2022-09-27T16:18:02.0723015Z Running distributed tests for the test backend with file init_method
2022-09-27T16:18:02.0734064Z Running distributed tests for the gloo backend with file init_method
2022-09-27T16:18:02.0740245Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:02.073697]
2022-09-27T16:18:02.0748744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:02.074524]
2022-09-27T16:18:03.2698144Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:18:03.2698865Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2699142Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2699432Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2699685Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2699958Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2700239Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:18:03.2700499Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:18:03.2785909Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:18:04.2759350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:04.2759737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:04.2761393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:04.2761754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:04.4276895Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:04.4285966Z 
2022-09-27T16:18:04.4286062Z Running tests...
2022-09-27T16:18:04.4286494Z ----------------------------------------------------------------------
2022-09-27T16:18:04.6639086Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12897
2022-09-27T16:18:04.6664994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12898
2022-09-27T16:18:04.6690770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12899
2022-09-27T16:18:05.7684050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:05.7684480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:05.7686921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:05.7687459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:05.7871095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:05.7871559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:05.7873918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:05.7874567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:05.8034290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:05.8035196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:05.8037524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:05.8038143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:05.8907215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:05.9087744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:05.9284433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:05.9494690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:05.9495517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:05.9496233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:18:05.9497246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:05.9498072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:05.9498952Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:06.2732137Z ok (1.844s)
2022-09-27T16:18:06.2732385Z 
2022-09-27T16:18:06.2732782Z ----------------------------------------------------------------------
2022-09-27T16:18:06.2733043Z Ran 1 test in 1.844s
2022-09-27T16:18:06.2733146Z 
2022-09-27T16:18:06.2733206Z OK
2022-09-27T16:18:06.2733315Z 
2022-09-27T16:18:06.2733406Z Generating XML reports...
2022-09-27T16:18:06.2767703Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161804.xml
2022-09-27T16:18:07.5649253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:07.5649630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:07.5650773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:07.5651131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:07.7156845Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:07.7165044Z 
2022-09-27T16:18:07.7165492Z Running tests...
2022-09-27T16:18:07.7166016Z ----------------------------------------------------------------------
2022-09-27T16:18:07.9491807Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12950
2022-09-27T16:18:07.9518614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12951
2022-09-27T16:18:07.9544801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12952
2022-09-27T16:18:09.0480996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:09.0481403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:09.0483968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:09.0484548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:09.0590118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:09.0590484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:09.0593201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:09.0593678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:09.0724679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:09.0725176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:09.0729374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:09.0729833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:09.1725043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:09.1829597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:09.1964359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:09.2133692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:09.2235349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:18:09.2236229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:09.2237333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:09.2238003Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:09.2238542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-09-27T16:18:09.5586198Z ok (1.842s)
2022-09-27T16:18:09.5587338Z 
2022-09-27T16:18:09.5587656Z ----------------------------------------------------------------------
2022-09-27T16:18:09.5587917Z Ran 1 test in 1.842s
2022-09-27T16:18:09.5588033Z 
2022-09-27T16:18:09.5588094Z OK
2022-09-27T16:18:09.5588187Z 
2022-09-27T16:18:09.5588280Z Generating XML reports...
2022-09-27T16:18:09.5621463Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161807.xml
2022-09-27T16:18:10.8457729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:10.8458147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:10.8460323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:10.8460697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:10.9967537Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:10.9976308Z 
2022-09-27T16:18:10.9976437Z Running tests...
2022-09-27T16:18:10.9977389Z ----------------------------------------------------------------------
2022-09-27T16:18:10.9986940Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:10.9987347Z 
2022-09-27T16:18:10.9987737Z ----------------------------------------------------------------------
2022-09-27T16:18:10.9988131Z Ran 1 test in 0.001s
2022-09-27T16:18:10.9988315Z 
2022-09-27T16:18:10.9988440Z OK (skipped=1)
2022-09-27T16:18:10.9988612Z 
2022-09-27T16:18:10.9988746Z Generating XML reports...
2022-09-27T16:18:11.0012910Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161810.xml
2022-09-27T16:18:12.1971700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:12.1972075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:12.1974051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:12.1974405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:12.3479061Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:12.3488199Z 
2022-09-27T16:18:12.3488285Z Running tests...
2022-09-27T16:18:12.3488677Z ----------------------------------------------------------------------
2022-09-27T16:18:12.3497308Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:12.3497930Z 
2022-09-27T16:18:12.3498393Z ----------------------------------------------------------------------
2022-09-27T16:18:12.3498831Z Ran 1 test in 0.001s
2022-09-27T16:18:12.3499038Z 
2022-09-27T16:18:12.3499174Z OK (skipped=1)
2022-09-27T16:18:12.3499317Z 
2022-09-27T16:18:12.3499409Z Generating XML reports...
2022-09-27T16:18:12.3522761Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161812.xml
2022-09-27T16:18:13.5388816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:13.5389208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:13.5391136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:13.5391501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:13.6899920Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:13.6909296Z 
2022-09-27T16:18:13.6909465Z Running tests...
2022-09-27T16:18:13.6909798Z ----------------------------------------------------------------------
2022-09-27T16:18:13.6918520Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:13.6918766Z 
2022-09-27T16:18:13.6919086Z ----------------------------------------------------------------------
2022-09-27T16:18:13.6919332Z Ran 1 test in 0.001s
2022-09-27T16:18:13.6919446Z 
2022-09-27T16:18:13.6919506Z OK (skipped=1)
2022-09-27T16:18:13.6919617Z 
2022-09-27T16:18:13.6919704Z Generating XML reports...
2022-09-27T16:18:13.6943034Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161813.xml
2022-09-27T16:18:14.8827502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:14.8828127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:14.8829499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:14.8830094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:15.0333662Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:18:15.0343180Z 
2022-09-27T16:18:15.0343498Z Running tests...
2022-09-27T16:18:15.0344420Z ----------------------------------------------------------------------
2022-09-27T16:18:15.0353465Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:15.0353865Z 
2022-09-27T16:18:15.0354229Z ----------------------------------------------------------------------
2022-09-27T16:18:15.0354629Z Ran 1 test in 0.001s
2022-09-27T16:18:15.0354806Z 
2022-09-27T16:18:15.0355194Z OK (skipped=1)
2022-09-27T16:18:15.0355389Z 
2022-09-27T16:18:15.0355526Z Generating XML reports...
2022-09-27T16:18:15.0379931Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161815.xml
2022-09-27T16:18:15.4943577Z Running distributed/test_c10d_object_collectives ... [2022-09-27 16:18:15.493978]
2022-09-27T16:18:15.4944456Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:15.494036]
2022-09-27T16:18:16.6754111Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2022-09-27T16:18:16.6765382Z 
2022-09-27T16:18:16.6765887Z Running tests...
2022-09-27T16:18:16.6766286Z ----------------------------------------------------------------------
2022-09-27T16:18:16.9101786Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13043
2022-09-27T16:18:16.9127710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13044
2022-09-27T16:18:17.9566754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:17.9567442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:17.9810937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:17.9811611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:17.9877945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:17.9878594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:18.0125813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:18.0126257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:18.1091104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:18.1334355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:18.1544099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:18.1544827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:18.1545462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:18.1545993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:18.4161219Z ok (1.739s)
2022-09-27T16:18:18.4211034Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13071
2022-09-27T16:18:18.4242078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13072
2022-09-27T16:18:19.4662230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:19.4662907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:19.4689675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:19.4690320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:19.4997792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:19.4998417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:19.5031550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:19.5032163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:19.6212250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:19.6234302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:19.6421799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:19.6422418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:19.6423259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:19.6423804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:19.9271967Z ok (1.511s)
2022-09-27T16:18:19.9323144Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13099
2022-09-27T16:18:19.9355911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13100
2022-09-27T16:18:20.9721231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:20.9721858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:21.0027983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:21.0028632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:21.0094320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:21.0094963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:21.0429994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:21.0430642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:21.1240706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:21.1636975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:21.1750740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:21.1751399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:21.1752345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:21.1753184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:21.4385922Z ok (1.511s)
2022-09-27T16:18:21.4427476Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13127
2022-09-27T16:18:21.4463389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13128
2022-09-27T16:18:22.4520868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:22.4521261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:22.4750591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:22.4750950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:22.4852789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:22.4853597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:22.5070919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:22.5071646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:22.6070358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:22.6305484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:22.6480306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:22.6480898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:22.6481783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:22.6482460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:22.9496347Z ok (1.511s)
2022-09-27T16:18:22.9496584Z 
2022-09-27T16:18:22.9497069Z ----------------------------------------------------------------------
2022-09-27T16:18:22.9497329Z Ran 4 tests in 6.273s
2022-09-27T16:18:22.9497457Z 
2022-09-27T16:18:22.9497518Z OK
2022-09-27T16:18:22.9497611Z 
2022-09-27T16:18:22.9497702Z Generating XML reports...
2022-09-27T16:18:22.9533995Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220927161816.xml
2022-09-27T16:18:23.2491297Z Running distributed/fsdp/test_fsdp_apply ... [2022-09-27 16:18:23.248761]
2022-09-27T16:18:23.2491850Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:23.248842]
2022-09-27T16:18:24.4516774Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-09-27T16:18:24.4528311Z 
2022-09-27T16:18:24.4528691Z Running tests...
2022-09-27T16:18:24.4529336Z ----------------------------------------------------------------------
2022-09-27T16:18:24.4535764Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-09-27T16:18:24.6894050Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13166
2022-09-27T16:18:24.6920164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13167
2022-09-27T16:18:25.7801360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:25.7802039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:25.7805098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:25.7805697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:25.8171917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:25.8172665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:25.8204526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:25.8205032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:25.9056754Z dist init r=0, world=2
2022-09-27T16:18:25.9422617Z dist init r=1, world=2
2022-09-27T16:18:25.9529954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:25.9530642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:25.9531243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:25.9531948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:25.9636084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:25.9636867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:26.1955420Z skip: Need at least 2 CUDA devices (1.742s)
2022-09-27T16:18:26.1963820Z   test_nested_module_apply (__main__.TestApply)
2022-09-27T16:18:26.2007577Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13194
2022-09-27T16:18:26.2041392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13195
2022-09-27T16:18:27.2705704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:27.2706114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:27.2708797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:27.2709313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:27.3094063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:27.3094488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:27.3097593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:27.3097964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:27.3942143Z dist init r=1, world=2
2022-09-27T16:18:27.4323618Z dist init r=0, world=2
2022-09-27T16:18:27.4452772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:27.4453211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:27.4453828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:27.4454365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:27.4458769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:27.4459177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:27.7073546Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:18:27.7078330Z   test_transformer_module_apply (__main__.TestApply)
2022-09-27T16:18:27.7122747Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13222
2022-09-27T16:18:27.7154989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13223
2022-09-27T16:18:28.8041619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:28.8042008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:28.8044798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:28.8045315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:28.8087615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:28.8088116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:28.8092144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:28.8092643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:28.9282395Z dist init r=1, world=2
2022-09-27T16:18:28.9325009Z dist init r=0, world=2
2022-09-27T16:18:28.9491024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:28.9491902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:28.9492985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:28.9493569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:28.9596055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:28.9596721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:29.2187801Z skip: Need at least 2 CUDA devices (1.511s)
2022-09-27T16:18:29.2188049Z 
2022-09-27T16:18:29.2188588Z ----------------------------------------------------------------------
2022-09-27T16:18:29.2188879Z Ran 3 tests in 4.766s
2022-09-27T16:18:29.2189004Z 
2022-09-27T16:18:29.2189084Z OK (skipped=3)
2022-09-27T16:18:29.2189194Z 
2022-09-27T16:18:29.2189279Z Generating XML reports...
2022-09-27T16:18:29.2223100Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220927161824.xml
2022-09-27T16:18:29.5008183Z Running distributed/_shard/checkpoint/test_checkpoint ... [2022-09-27 16:18:29.500376]
2022-09-27T16:18:29.5008774Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:29.500450]
2022-09-27T16:18:30.7013099Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint
2022-09-27T16:18:30.7025636Z 
2022-09-27T16:18:30.7025781Z Running tests...
2022-09-27T16:18:30.7026636Z ----------------------------------------------------------------------
2022-09-27T16:18:30.7040077Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:30.7045894Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:30.7051436Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:18:30.7056137Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:18:30.7066997Z   test_load_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:30.9436623Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13261
2022-09-27T16:18:30.9462879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13262
2022-09-27T16:18:30.9490360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13263
2022-09-27T16:18:30.9520856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13264
2022-09-27T16:18:32.1489184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:32.1489779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:32.1493313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:32.1494010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:32.1539981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:32.1540823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:32.1544919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:32.1545606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:32.1546406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:32.1547031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:32.1550736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:32.1551361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:32.1950042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:32.1950735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:32.1954205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:32.1954827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:32.2781239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:18:32.2787725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:32.2794956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:32.3191243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:32.6564905Z ok (1.950s)
2022-09-27T16:18:32.6575597Z   test_save_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:18:32.6627056Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13305
2022-09-27T16:18:32.6660359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13306
2022-09-27T16:18:32.6688470Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13307
2022-09-27T16:18:32.6717891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13308
2022-09-27T16:18:33.7926226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:33.7926782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:33.7930250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:33.7930780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:33.8265210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:33.8265852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:33.8269127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:33.8269750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:33.8291903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:33.8293018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:33.8296550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:33.8297272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:33.8490224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:33.8491131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:33.8495179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:33.8495986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:33.9186031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:33.9550131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:33.9551152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:33.9731830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:18:34.2755679Z ok (1.618s)
2022-09-27T16:18:34.2755934Z 
2022-09-27T16:18:34.2756490Z ----------------------------------------------------------------------
2022-09-27T16:18:34.2756806Z Ran 8 tests in 3.573s
2022-09-27T16:18:34.2756912Z 
2022-09-27T16:18:34.2757006Z OK (skipped=6)
2022-09-27T16:18:34.2757116Z 
2022-09-27T16:18:34.2757204Z Generating XML reports...
2022-09-27T16:18:34.2796659Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220927161830.xml
2022-09-27T16:18:34.2800204Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220927161830.xml
2022-09-27T16:18:34.5631057Z Running distributed/fsdp/test_fsdp_memory ... [2022-09-27 16:18:34.562676]
2022-09-27T16:18:34.5631660Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:34.562776]
2022-09-27T16:18:35.7414231Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-09-27T16:18:35.7425489Z 
2022-09-27T16:18:35.7425818Z Running tests...
2022-09-27T16:18:35.7426475Z ----------------------------------------------------------------------
2022-09-27T16:18:35.9828124Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13360
2022-09-27T16:18:35.9854637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13361
2022-09-27T16:18:37.0701948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:37.0702370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:37.0706236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:37.0706793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:37.0749597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:37.0750105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:37.0754082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:37.0754452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:37.1977339Z dist init r=0, world=2
2022-09-27T16:18:37.1981594Z dist init r=1, world=2
2022-09-27T16:18:37.2186304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:37.2186895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:37.2187920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:37.2188601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:37.2292441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:37.2292839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:37.4888725Z skip: Need at least 2 CUDA devices (1.746s)
2022-09-27T16:18:37.4951560Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13388
2022-09-27T16:18:37.4984372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13389
2022-09-27T16:18:38.5644083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:38.5644732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:38.5646651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:38.5647209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:38.5821986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:38.5822366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:38.5826372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:38.5826880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:38.6880238Z dist init r=0, world=2
2022-09-27T16:18:38.7085568Z dist init r=1, world=2
2022-09-27T16:18:38.7290362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:38.7290804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:38.7291419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:38.7292148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:38.7296399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:38.7296815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:39.0015709Z skip: Need at least 2 CUDA devices (1.512s)
2022-09-27T16:18:39.0016029Z 
2022-09-27T16:18:39.0016467Z ----------------------------------------------------------------------
2022-09-27T16:18:39.0016745Z Ran 2 tests in 3.259s
2022-09-27T16:18:39.0016861Z 
2022-09-27T16:18:39.0016934Z OK (skipped=2)
2022-09-27T16:18:39.0017042Z 
2022-09-27T16:18:39.0017128Z Generating XML reports...
2022-09-27T16:18:39.0050173Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220927161835.xml
2022-09-27T16:18:39.2966066Z Running distributed/fsdp/test_fsdp_input ... [2022-09-27 16:18:39.296226]
2022-09-27T16:18:39.2966651Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:39.296305]
2022-09-27T16:18:40.4969777Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-09-27T16:18:40.4981326Z 
2022-09-27T16:18:40.4981677Z Running tests...
2022-09-27T16:18:40.4982313Z ----------------------------------------------------------------------
2022-09-27T16:18:40.4996414Z   test_input_type_dict (__main__.TestInput)
2022-09-27T16:18:40.7336869Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13427
2022-09-27T16:18:41.7913783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:41.7914155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:41.7918387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:41.7918834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:41.9143067Z dist init r=0, world=1
2022-09-27T16:18:41.9151288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:41.9151998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:18:41.9155993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:42.1366513Z skip: Need at least 1 CUDA device (1.638s)
2022-09-27T16:18:42.1380384Z   test_input_type_list (__main__.TestInput)
2022-09-27T16:18:42.1426729Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13441
2022-09-27T16:18:43.2059691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:43.2060054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:43.2063252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:43.2063614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:43.3288784Z dist init r=0, world=1
2022-09-27T16:18:43.3297421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:43.3298032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:18:43.3301462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:43.5454775Z skip: Need at least 1 CUDA device (1.409s)
2022-09-27T16:18:43.5455041Z 
2022-09-27T16:18:43.5455537Z ----------------------------------------------------------------------
2022-09-27T16:18:43.5455966Z Ran 2 tests in 3.047s
2022-09-27T16:18:43.5456100Z 
2022-09-27T16:18:43.5456172Z OK (skipped=2)
2022-09-27T16:18:43.5456278Z 
2022-09-27T16:18:43.5456365Z Generating XML reports...
2022-09-27T16:18:43.5489616Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220927161840.xml
2022-09-27T16:18:43.8393021Z Running distributed/rpc/test_share_memory ... [2022-09-27 16:18:43.838899]
2022-09-27T16:18:43.8393840Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:43.838982]
2022-09-27T16:18:44.9891211Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:18:44.9891772Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-09-27T16:18:44.9892048Z test_case (__main__.TestRPCPickler)
2022-09-27T16:18:46.0107353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:46.0107728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:46.0115005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:46.0115370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:46.1733125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-09-27T16:18:46.1742055Z 
2022-09-27T16:18:46.1742163Z Running tests...
2022-09-27T16:18:46.1742648Z ----------------------------------------------------------------------
2022-09-27T16:18:47.4361047Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:47.4361686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:47.4648952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:47.4649495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:48.0357136Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f923cdfab90>
2022-09-27T16:18:48.0357734Z Traceback (most recent call last):
2022-09-27T16:18:48.0358313Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-09-27T16:18:48.0358771Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref
2022-09-27T16:18:48.0359138Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-09-27T16:18:48.0359439Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f923cdfab90>
2022-09-27T16:18:48.0359700Z Traceback (most recent call last):
2022-09-27T16:18:48.0360082Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-09-27T16:18:48.0360511Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref
2022-09-27T16:18:48.0360889Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-09-27T16:18:48.1521306Z ok (1.978s)
2022-09-27T16:18:48.1521552Z 
2022-09-27T16:18:48.1521866Z ----------------------------------------------------------------------
2022-09-27T16:18:48.1522161Z Ran 1 test in 1.978s
2022-09-27T16:18:48.1522288Z 
2022-09-27T16:18:48.1522350Z OK
2022-09-27T16:18:48.1522442Z 
2022-09-27T16:18:48.1522532Z Generating XML reports...
2022-09-27T16:18:48.1555860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220927161846.xml
2022-09-27T16:18:48.6579229Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-09-27 16:18:48.657486]
2022-09-27T16:18:48.6579837Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:48.657565]
2022-09-27T16:18:49.8773943Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-09-27T16:18:49.8784469Z 
2022-09-27T16:18:49.8784597Z Running tests...
2022-09-27T16:18:49.8785314Z ----------------------------------------------------------------------
2022-09-27T16:18:50.1150793Z   test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13583
2022-09-27T16:18:50.1176564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13584
2022-09-27T16:18:50.1204424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13585
2022-09-27T16:18:50.1233384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13586
2022-09-27T16:18:51.3417379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:51.3417932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:51.3421987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:51.3422549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.3699528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:51.3700123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:51.3702930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:51.3703558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.3791903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:51.3792785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:51.3797037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:51.3797826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.4262743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:51.4263340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:51.4266826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:51.4267512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.4677918Z dist init r=3, world=4
2022-09-27T16:18:51.4948472Z dist init r=2, world=4
2022-09-27T16:18:51.5049895Z dist init r=1, world=4
2022-09-27T16:18:51.5529947Z dist init r=0, world=4
2022-09-27T16:18:51.5761455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:51.5862252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:51.5964264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:18:51.5965196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:18:51.5966153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:51.5975414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:51.5976254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:51.5977106Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:51.5977725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:51.5978279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:51.5978858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:18:51.5979509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:51.9281043Z skip: Need at least 2 CUDA devices (2.049s)
2022-09-27T16:18:51.9281354Z 
2022-09-27T16:18:51.9281791Z ----------------------------------------------------------------------
2022-09-27T16:18:51.9282066Z Ran 1 test in 2.050s
2022-09-27T16:18:51.9282184Z 
2022-09-27T16:18:51.9282257Z OK (skipped=1)
2022-09-27T16:18:51.9282353Z 
2022-09-27T16:18:51.9282437Z Generating XML reports...
2022-09-27T16:18:51.9317346Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220927161849.xml
2022-09-27T16:18:52.2195631Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-09-27 16:18:52.219084]
2022-09-27T16:18:52.2196670Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:52.219160]
2022-09-27T16:18:53.4245315Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-09-27T16:18:53.4255947Z 
2022-09-27T16:18:53.4256047Z Running tests...
2022-09-27T16:18:53.4256638Z ----------------------------------------------------------------------
2022-09-27T16:18:53.4267217Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-09-27T16:18:53.6632125Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13650
2022-09-27T16:18:53.6657601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13651
2022-09-27T16:18:53.6684480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13652
2022-09-27T16:18:53.6713372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13653
2022-09-27T16:18:54.8394579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.8395223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.8398939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.8399501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.8502326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.8502982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.8507011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.8507647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.8625120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.8625742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.8630237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.8630848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.8754003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.8754546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.8759786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.8760446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.9690456Z dist init r=0, world=4
2022-09-27T16:18:54.9760128Z dist init r=1, world=4
2022-09-27T16:18:54.9874678Z dist init r=2, world=4
2022-09-27T16:18:54.9998664Z dist init r=3, world=4
2022-09-27T16:18:55.0106389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:55.0271985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:55.0373560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:18:55.0374249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:18:55.0375112Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:55.0375650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:55.0376182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:55.0411004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:18:55.0480820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:55.0481570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:55.0482129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:55.0482663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:18:55.3759267Z skip: Need at least 2 CUDA devices (1.950s)
2022-09-27T16:18:55.3759463Z 
2022-09-27T16:18:55.3759880Z ----------------------------------------------------------------------
2022-09-27T16:18:55.3760230Z Ran 1 test in 1.950s
2022-09-27T16:18:55.3760384Z 
2022-09-27T16:18:55.3760478Z OK (skipped=1)
2022-09-27T16:18:55.3760574Z 
2022-09-27T16:18:55.3760661Z Generating XML reports...
2022-09-27T16:18:55.3796119Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220927161853.xml
2022-09-27T16:18:55.6632236Z Running distributed/fsdp/test_fsdp_traversal ... [2022-09-27 16:18:55.662776]
2022-09-27T16:18:55.6633048Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:55.662857]
2022-09-27T16:18:56.8913842Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-09-27T16:18:56.8925625Z 
2022-09-27T16:18:56.8926079Z Running tests...
2022-09-27T16:18:56.8926482Z ----------------------------------------------------------------------
2022-09-27T16:18:57.1335963Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13717
2022-09-27T16:18:57.1362542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13718
2022-09-27T16:18:58.2409319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:58.2409699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:58.2412965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:58.2413341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:58.2571687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:58.2572344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:58.2575411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:58.2576037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:58.3750860Z dist init r=0, world=2
2022-09-27T16:18:58.3849886Z dist init r=1, world=2
2022-09-27T16:18:58.3960481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:58.3961199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:58.3961913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:58.3962469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:58.3965535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:58.3965919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:58.6399949Z skip: Need at least 2 CUDA devices (1.747s)
2022-09-27T16:18:58.6400273Z 
2022-09-27T16:18:58.6400787Z ----------------------------------------------------------------------
2022-09-27T16:18:58.6401077Z Ran 1 test in 1.747s
2022-09-27T16:18:58.6401179Z 
2022-09-27T16:18:58.6401254Z OK (skipped=1)
2022-09-27T16:18:58.6401363Z 
2022-09-27T16:18:58.6401452Z Generating XML reports...
2022-09-27T16:18:58.6436258Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220927161856.xml
2022-09-27T16:18:58.9279899Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-09-27 16:18:58.927567]
2022-09-27T16:18:58.9280760Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:18:58.927669]
2022-09-27T16:19:00.1060499Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper
2022-09-27T16:19:00.1074129Z 
2022-09-27T16:19:00.1074450Z Running tests...
2022-09-27T16:19:00.3372624Z ----------------------------------------------------------------------
2022-09-27T16:19:00.3373220Z   test_empty_module (__main__.TestFlattenParams) ... ok (0.230s)
2022-09-27T16:19:00.3438885Z   test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.007s)
2022-09-27T16:19:00.3511616Z   test_num_params (__main__.TestFlattenParams) ... ok (0.007s)
2022-09-27T16:19:00.3722985Z   test_output (__main__.TestFlattenParams) ... ok (0.021s)
2022-09-27T16:19:00.3820972Z   test_partial_flattening (__main__.TestFlattenParams) ... ok (0.010s)
2022-09-27T16:19:00.3893534Z   test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.007s)
2022-09-27T16:19:00.3964166Z   test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.007s)
2022-09-27T16:19:00.4189446Z   test_shared_params_output (__main__.TestFlattenParams) ... ok (0.022s)
2022-09-27T16:19:00.4525148Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.033s)
2022-09-27T16:19:00.4531445Z   test_empty_module (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s)
2022-09-27T16:19:00.4535612Z   test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4538681Z   test_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4540342Z   test_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4560260Z   test_partial_flattening (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.002s)
2022-09-27T16:19:00.4594895Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.003s)
2022-09-27T16:19:00.4597425Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4599785Z   test_shared_params_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4604816Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4610210Z   test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4613856Z   test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4616225Z   test_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4618628Z   test_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4633971Z   test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.002s)
2022-09-27T16:19:00.4667648Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.003s)
2022-09-27T16:19:00.4670219Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4672500Z   test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4678343Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-09-27T16:19:00.4678725Z 
2022-09-27T16:19:00.4679234Z ----------------------------------------------------------------------
2022-09-27T16:19:00.4679500Z Ran 27 tests in 0.360s
2022-09-27T16:19:00.4679620Z 
2022-09-27T16:19:00.4679695Z OK (skipped=18)
2022-09-27T16:19:00.4679805Z 
2022-09-27T16:19:00.4679878Z Generating XML reports...
2022-09-27T16:19:00.4708805Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220927161900.xml
2022-09-27T16:19:00.4718420Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220927161900.xml
2022-09-27T16:19:00.4728714Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220927161900.xml
2022-09-27T16:19:00.7223334Z Running distributed/elastic/utils/util_test ... [2022-09-27 16:19:00.721891]
2022-09-27T16:19:00.7224064Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:00.721965]
2022-09-27T16:19:01.8780541Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-09-27T16:19:01.8791890Z 
2022-09-27T16:19:01.8792006Z Running tests...
2022-09-27T16:19:01.8792562Z ----------------------------------------------------------------------
2022-09-27T16:19:02.1102697Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.231s)
2022-09-27T16:19:02.1118556Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-09-27T16:19:02.1137852Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.002s)
2022-09-27T16:19:02.1918963Z   test_get_logger (__main__.UtilTest) ... ok (0.078s)
2022-09-27T16:19:02.1925377Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:19:02.1934095Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:19:02.1945747Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:19:02.1945999Z 
2022-09-27T16:19:02.1946298Z ----------------------------------------------------------------------
2022-09-27T16:19:02.1946549Z Ran 7 tests in 0.315s
2022-09-27T16:19:02.1946665Z 
2022-09-27T16:19:02.1946726Z OK
2022-09-27T16:19:02.1946817Z 
2022-09-27T16:19:02.1946894Z Generating XML reports...
2022-09-27T16:19:02.1973867Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220927161901.xml
2022-09-27T16:19:02.1978581Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220927161901.xml
2022-09-27T16:19:02.4445169Z Running distributed/_shard/checkpoint/test_file_system_checkpoint ... [2022-09-27 16:19:02.444124]
2022-09-27T16:19:02.4445802Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:02.444202]
2022-09-27T16:19:03.6250854Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint
2022-09-27T16:19:03.6264776Z 
2022-09-27T16:19:03.6265253Z Running tests...
2022-09-27T16:19:03.6265678Z ----------------------------------------------------------------------
2022-09-27T16:19:03.6278188Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:03.6300316Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-09-27T16:19:03.6307498Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:03.6324573Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-09-27T16:19:03.8792199Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.247s)
2022-09-27T16:19:03.8803155Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:03.8803742Z 
2022-09-27T16:19:03.8804065Z ----------------------------------------------------------------------
2022-09-27T16:19:03.8804472Z Ran 6 tests in 0.254s
2022-09-27T16:19:03.8804592Z 
2022-09-27T16:19:03.8804666Z OK (skipped=5)
2022-09-27T16:19:03.8804777Z 
2022-09-27T16:19:03.8804866Z Generating XML reports...
2022-09-27T16:19:03.8827109Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220927161903.xml
2022-09-27T16:19:03.8832777Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220927161903.xml
2022-09-27T16:19:03.8835700Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927161903.xml
2022-09-27T16:19:04.1309472Z Running distributed/elastic/utils/logging_test ... [2022-09-27 16:19:04.130575]
2022-09-27T16:19:04.1310078Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:04.130647]
2022-09-27T16:19:05.3443781Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-09-27T16:19:05.3454956Z 
2022-09-27T16:19:05.3455415Z Running tests...
2022-09-27T16:19:05.3455807Z ----------------------------------------------------------------------
2022-09-27T16:19:05.6002609Z   test_derive_module_name (__main__.LoggingTest) ... ok (0.254s)
2022-09-27T16:19:05.6021265Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-09-27T16:19:05.6021623Z 
2022-09-27T16:19:05.6022144Z ----------------------------------------------------------------------
2022-09-27T16:19:05.6022430Z Ran 2 tests in 0.257s
2022-09-27T16:19:05.6022556Z 
2022-09-27T16:19:05.6022619Z OK
2022-09-27T16:19:05.6022711Z 
2022-09-27T16:19:05.6022793Z Generating XML reports...
2022-09-27T16:19:05.6047186Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220927161905.xml
2022-09-27T16:19:05.8397046Z Running distributed/fsdp/test_utils ... [2022-09-27 16:19:05.839361]
2022-09-27T16:19:05.8397597Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:05.839443]
2022-09-27T16:19:07.0360482Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-09-27T16:19:07.0371525Z 
2022-09-27T16:19:07.0371819Z Running tests...
2022-09-27T16:19:07.0372438Z ----------------------------------------------------------------------
2022-09-27T16:19:07.2651335Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.228s)
2022-09-27T16:19:07.2686730Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.004s)
2022-09-27T16:19:07.2707829Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2022-09-27T16:19:07.2714418Z   test_packed_sequence (__main__.TestUtils)
2022-09-27T16:19:07.2735020Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-09-27T16:19:07.2746288Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-09-27T16:19:07.2746594Z 
2022-09-27T16:19:07.2746985Z ----------------------------------------------------------------------
2022-09-27T16:19:07.2747382Z Ran 5 tests in 0.237s
2022-09-27T16:19:07.2747569Z 
2022-09-27T16:19:07.2747694Z OK (skipped=2)
2022-09-27T16:19:07.2747858Z 
2022-09-27T16:19:07.2748003Z Generating XML reports...
2022-09-27T16:19:07.2775369Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220927161907.xml
2022-09-27T16:19:07.5228006Z Running distributed/nn/jit/test_instantiator ... [2022-09-27 16:19:07.522412]
2022-09-27T16:19:07.5228598Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:07.522494]
2022-09-27T16:19:08.6852929Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-09-27T16:19:08.6863786Z 
2022-09-27T16:19:08.6864445Z Running tests...
2022-09-27T16:19:08.6865054Z ----------------------------------------------------------------------
2022-09-27T16:19:08.9119078Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.225s)
2022-09-27T16:19:08.9138427Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-09-27T16:19:08.9206808Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.007s)
2022-09-27T16:19:08.9207178Z 
2022-09-27T16:19:08.9207715Z ----------------------------------------------------------------------
2022-09-27T16:19:08.9208005Z Ran 3 tests in 0.234s
2022-09-27T16:19:08.9208122Z 
2022-09-27T16:19:08.9208184Z OK
2022-09-27T16:19:08.9208276Z 
2022-09-27T16:19:08.9208362Z Generating XML reports...
2022-09-27T16:19:08.9232357Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220927161908.xml
2022-09-27T16:19:09.1701170Z Running distributed/elastic/metrics/api_test ... [2022-09-27 16:19:09.169790]
2022-09-27T16:19:09.1701764Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:09.169869]
2022-09-27T16:19:10.3454893Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-09-27T16:19:10.3466750Z 
2022-09-27T16:19:10.3467053Z Running tests...
2022-09-27T16:19:10.3467688Z ----------------------------------------------------------------------
2022-09-27T16:19:10.5722557Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (0.225s)
2022-09-27T16:19:10.5734877Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2022-09-27T16:19:10.5752305Z   test_profile (__main__.MetricsApiTest) ... ok (0.002s)
2022-09-27T16:19:10.5752932Z 
2022-09-27T16:19:10.5753330Z ----------------------------------------------------------------------
2022-09-27T16:19:10.5753677Z Ran 3 tests in 0.229s
2022-09-27T16:19:10.5753793Z 
2022-09-27T16:19:10.5753859Z OK
2022-09-27T16:19:10.5753949Z 
2022-09-27T16:19:10.5754033Z Generating XML reports...
2022-09-27T16:19:10.5778146Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220927161910.xml
2022-09-27T16:19:10.8222116Z Running distributed/test_data_parallel ... [2022-09-27 16:19:10.821824]
2022-09-27T16:19:10.8222877Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:10.821897]
2022-09-27T16:19:12.2350336Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-09-27T16:19:12.2363661Z 
2022-09-27T16:19:12.2363798Z Running tests...
2022-09-27T16:19:12.2364444Z ----------------------------------------------------------------------
2022-09-27T16:19:12.2372041Z   test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2383279Z   test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2392029Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2400531Z   test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s)
2022-09-27T16:19:12.2409021Z   test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2419173Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2423456Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2453238Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-09-27T16:19:12.2453739Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s)
2022-09-27T16:19:12.2460476Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2469128Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2491543Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2022-09-27T16:19:12.2498512Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2513204Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2519474Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2531426Z   test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2535885Z   test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2547696Z   test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2549721Z   test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2555081Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2557274Z   test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2565666Z   test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2574481Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2579691Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.000s)
2022-09-27T16:19:12.2593680Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2600933Z   test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2607501Z   test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2613398Z   test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2616087Z   test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2618781Z   test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-09-27T16:19:12.2652630Z   test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s)
2022-09-27T16:19:12.2658941Z   test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-09-27T16:19:12.2676048Z   test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.002s)
2022-09-27T16:19:12.2686368Z   test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2695933Z   test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2708215Z   test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2720029Z   test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2731735Z   test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2744415Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2757045Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2769135Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2781283Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2793711Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2805901Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2818089Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2830606Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2843422Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-09-27T16:19:12.2843840Z 
2022-09-27T16:19:12.2844143Z ----------------------------------------------------------------------
2022-09-27T16:19:12.2844379Z Ran 46 tests in 0.048s
2022-09-27T16:19:12.2844496Z 
2022-09-27T16:19:12.2844568Z OK (skipped=46)
2022-09-27T16:19:12.2844680Z 
2022-09-27T16:19:12.2844764Z Generating XML reports...
2022-09-27T16:19:12.2896481Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220927161912.xml
2022-09-27T16:19:12.2912032Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220927161912.xml
2022-09-27T16:19:12.5307049Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-09-27 16:19:12.530237]
2022-09-27T16:19:12.5307918Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:12.530310]
2022-09-27T16:19:13.7039905Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-09-27T16:19:13.7051867Z 
2022-09-27T16:19:13.7051969Z Running tests...
2022-09-27T16:19:13.7052514Z ----------------------------------------------------------------------
2022-09-27T16:19:13.7058869Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7069821Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7077654Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7081439Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:13.7090022Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7096848Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7107089Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7112046Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:13.7120678Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7129149Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7136423Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:13.7136818Z 
2022-09-27T16:19:13.7137129Z ----------------------------------------------------------------------
2022-09-27T16:19:13.7137390Z Ran 11 tests in 0.008s
2022-09-27T16:19:13.7137508Z 
2022-09-27T16:19:13.7137584Z OK (skipped=11)
2022-09-27T16:19:13.7137696Z 
2022-09-27T16:19:13.7137783Z Generating XML reports...
2022-09-27T16:19:13.7170431Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220927161913.xml
2022-09-27T16:19:13.9011875Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-09-27 16:19:13.900800]
2022-09-27T16:19:13.9012494Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:13.900872]
2022-09-27T16:19:15.0646687Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2022-09-27T16:19:15.0657007Z 
2022-09-27T16:19:15.0657276Z Running tests...
2022-09-27T16:19:15.0657896Z ----------------------------------------------------------------------
2022-09-27T16:19:15.0667154Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:15.0683785Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-09-27T16:19:15.0691209Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:15.0702402Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:15.0736480Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-09-27T16:19:15.0736898Z 
2022-09-27T16:19:15.0737288Z ----------------------------------------------------------------------
2022-09-27T16:19:15.0737697Z Ran 5 tests in 0.008s
2022-09-27T16:19:15.0737882Z 
2022-09-27T16:19:15.0738001Z OK (skipped=5)
2022-09-27T16:19:15.0738182Z 
2022-09-27T16:19:15.0738333Z Generating XML reports...
2022-09-27T16:19:15.0765542Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20220927161915.xml
2022-09-27T16:19:15.2645924Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-09-27 16:19:15.264216]
2022-09-27T16:19:15.2646557Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:15.264289]
2022-09-27T16:19:16.4341896Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-09-27T16:19:16.4352306Z 
2022-09-27T16:19:16.4352617Z Running tests...
2022-09-27T16:19:16.4352989Z ----------------------------------------------------------------------
2022-09-27T16:19:16.4367273Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:16.4395042Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-09-27T16:19:16.4406511Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:16.4406866Z 
2022-09-27T16:19:16.4407583Z ----------------------------------------------------------------------
2022-09-27T16:19:16.4407882Z Ran 3 tests in 0.005s
2022-09-27T16:19:16.4407999Z 
2022-09-27T16:19:16.4408073Z OK (skipped=3)
2022-09-27T16:19:16.4408262Z 
2022-09-27T16:19:16.4408349Z Generating XML reports...
2022-09-27T16:19:16.4433405Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220927161916.xml
2022-09-27T16:19:16.6272215Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-09-27 16:19:16.626855]
2022-09-27T16:19:16.6272931Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:16.626925]
2022-09-27T16:19:17.7993472Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-09-27T16:19:17.8005009Z 
2022-09-27T16:19:17.8005143Z Running tests...
2022-09-27T16:19:17.8005592Z ----------------------------------------------------------------------
2022-09-27T16:19:17.8013335Z   test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:17.8019176Z   test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:17.8026303Z   test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:17.8035096Z   test_inplace_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:17.8042206Z   test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:17.8042556Z 
2022-09-27T16:19:17.8042989Z ----------------------------------------------------------------------
2022-09-27T16:19:17.8043400Z Ran 5 tests in 0.004s
2022-09-27T16:19:17.8043520Z 
2022-09-27T16:19:17.8043601Z OK (skipped=5)
2022-09-27T16:19:17.8043709Z 
2022-09-27T16:19:17.8043794Z Generating XML reports...
2022-09-27T16:19:17.8070072Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220927161917.xml
2022-09-27T16:19:17.9908385Z Running distributed/_shard/test_partial_tensor ... [2022-09-27 16:19:17.990404]
2022-09-27T16:19:17.9909000Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:17.990484]
2022-09-27T16:19:19.1680392Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2022-09-27T16:19:19.1692592Z 
2022-09-27T16:19:19.1693057Z Running tests...
2022-09-27T16:19:19.1693478Z ----------------------------------------------------------------------
2022-09-27T16:19:19.1705007Z   test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:19.1712609Z   test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:19.1717156Z   test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:19.1727269Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:19.1738558Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:19.1738812Z 
2022-09-27T16:19:19.1739092Z ----------------------------------------------------------------------
2022-09-27T16:19:19.1739386Z Ran 5 tests in 0.005s
2022-09-27T16:19:19.1739489Z 
2022-09-27T16:19:19.1739567Z OK (skipped=5)
2022-09-27T16:19:19.1739675Z 
2022-09-27T16:19:19.1739761Z Generating XML reports...
2022-09-27T16:19:19.1764244Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220927161919.xml
2022-09-27T16:19:19.1767590Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220927161919.xml
2022-09-27T16:19:19.3624747Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-09-27 16:19:19.362035]
2022-09-27T16:19:19.3625398Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:19.362109]
2022-09-27T16:19:20.5209842Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2022-09-27T16:19:20.5220428Z 
2022-09-27T16:19:20.5220558Z Running tests...
2022-09-27T16:19:20.5221155Z ----------------------------------------------------------------------
2022-09-27T16:19:20.5229498Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:19:20.5230485Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5235674Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:19:20.5236115Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5241564Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:19:20.5242109Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5247619Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:19:20.5248061Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5253266Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:19:20.5253776Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5265173Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:20.5265544Z 
2022-09-27T16:19:20.5265783Z ----------------------------------------------------------------------
2022-09-27T16:19:20.5266032Z Ran 6 tests in 0.004s
2022-09-27T16:19:20.5266147Z 
2022-09-27T16:19:20.5266224Z OK (skipped=6)
2022-09-27T16:19:20.5266333Z 
2022-09-27T16:19:20.5266403Z Generating XML reports...
2022-09-27T16:19:20.5294004Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220927161920.xml
2022-09-27T16:19:20.7094455Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-09-27 16:19:20.709077]
2022-09-27T16:19:20.7095137Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:20.709154]
2022-09-27T16:19:21.8834069Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-09-27T16:19:21.8844416Z 
2022-09-27T16:19:21.8844496Z Running tests...
2022-09-27T16:19:21.8844986Z ----------------------------------------------------------------------
2022-09-27T16:19:21.8855599Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:21.8871886Z   test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-09-27T16:19:21.8872223Z 
2022-09-27T16:19:21.8872883Z ----------------------------------------------------------------------
2022-09-27T16:19:21.8873312Z Ran 2 tests in 0.003s
2022-09-27T16:19:21.8873434Z 
2022-09-27T16:19:21.8873508Z OK (skipped=2)
2022-09-27T16:19:21.8873617Z 
2022-09-27T16:19:21.8873804Z Generating XML reports...
2022-09-27T16:19:21.8897136Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220927161921.xml
2022-09-27T16:19:22.0793136Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-09-27 16:19:22.078935]
2022-09-27T16:19:22.0793753Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:22.079011]
2022-09-27T16:19:23.2534386Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-09-27T16:19:23.2545415Z 
2022-09-27T16:19:23.2545906Z Running tests...
2022-09-27T16:19:23.2546412Z ----------------------------------------------------------------------
2022-09-27T16:19:23.2553385Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:23.2560024Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:23.2560508Z 
2022-09-27T16:19:23.2560904Z ----------------------------------------------------------------------
2022-09-27T16:19:23.2561312Z Ran 2 tests in 0.001s
2022-09-27T16:19:23.2561516Z 
2022-09-27T16:19:23.2561662Z OK (skipped=2)
2022-09-27T16:19:23.2561849Z 
2022-09-27T16:19:23.2561944Z Generating XML reports...
2022-09-27T16:19:23.2586176Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220927161923.xml
2022-09-27T16:19:23.4499911Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-09-27 16:19:23.449556]
2022-09-27T16:19:23.4500591Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:23.449632]
2022-09-27T16:19:24.6476429Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-09-27T16:19:24.6486637Z 
2022-09-27T16:19:24.6486787Z Running tests...
2022-09-27T16:19:24.6487186Z ----------------------------------------------------------------------
2022-09-27T16:19:24.6505899Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-09-27T16:19:24.6506443Z 
2022-09-27T16:19:24.6506782Z ----------------------------------------------------------------------
2022-09-27T16:19:24.6507034Z Ran 1 test in 0.002s
2022-09-27T16:19:24.6507150Z 
2022-09-27T16:19:24.6507214Z OK (skipped=1)
2022-09-27T16:19:24.6507322Z 
2022-09-27T16:19:24.6507409Z Generating XML reports...
2022-09-27T16:19:24.6530501Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220927161924.xml
2022-09-27T16:19:24.8374191Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-09-27 16:19:24.837068]
2022-09-27T16:19:24.8374796Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:24.837141]
2022-09-27T16:19:26.0057038Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-09-27T16:19:26.0068239Z 
2022-09-27T16:19:26.0068347Z Running tests...
2022-09-27T16:19:26.0068795Z ----------------------------------------------------------------------
2022-09-27T16:19:26.0078456Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-09-27T16:19:26.0079071Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-09-27T16:19:26.0082134Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:26.0085797Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-09-27T16:19:26.0086450Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:26.0089349Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-09-27T16:19:26.0089657Z 
2022-09-27T16:19:26.0089912Z ----------------------------------------------------------------------
2022-09-27T16:19:26.0090165Z Ran 4 tests in 0.002s
2022-09-27T16:19:26.0090279Z 
2022-09-27T16:19:26.0090384Z OK (skipped=4)
2022-09-27T16:19:26.0090515Z 
2022-09-27T16:19:26.0090588Z Generating XML reports...
2022-09-27T16:19:26.0116368Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220927161926.xml
2022-09-27T16:19:26.1962565Z Running distributed/_shard/test_replicated_tensor ... [2022-09-27 16:19:26.195840]
2022-09-27T16:19:26.1963164Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:26.195914]
2022-09-27T16:19:27.5501893Z Running distributed/elastic/events/lib_test ... [2022-09-27 16:19:27.549799]
2022-09-27T16:19:27.5502411Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-09-27 16:19:27.549873]
2022-09-27T16:19:28.2013413Z ============================= test session starts ==============================
2022-09-27T16:19:28.2014131Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:28.2104210Z cachedir: .pytest_cache
2022-09-27T16:19:28.2104881Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:28.2105268Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:28.2105688Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:29.2935129Z collecting ... 
2022-09-27T16:19:29.2944708Z collecting 3 items                                                             
2022-09-27T16:19:29.2945039Z collected 8 items                                                              
2022-09-27T16:19:29.2948407Z 
2022-09-27T16:19:29.2960759Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2022-09-27T16:19:29.2970947Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2022-09-27T16:19:29.2983073Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2022-09-27T16:19:29.3693898Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2022-09-27T16:19:29.3707574Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2022-09-27T16:19:29.3716454Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2022-09-27T16:19:29.3726446Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2022-09-27T16:19:29.3740405Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2022-09-27T16:19:29.3741795Z 
2022-09-27T16:19:29.3742130Z ============================== 8 passed in 1.17s ===============================
2022-09-27T16:19:29.5744103Z Running distributed/fsdp/test_shard_utils ... [2022-09-27 16:19:29.574030]
2022-09-27T16:19:29.5744768Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:19:29.574109]
2022-09-27T16:19:30.9344146Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-09-27 16:19:30.934022]
2022-09-27T16:19:30.9344743Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-09-27 16:19:30.934102]
2022-09-27T16:19:32.2894948Z ============================= test session starts ==============================
2022-09-27T16:19:32.2895394Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:32.2966381Z cachedir: .pytest_cache
2022-09-27T16:19:32.2967097Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:32.2967454Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:32.2967703Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:32.2968100Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:32.3119233Z collecting ... 
2022-09-27T16:19:32.3119617Z collected 13 items                                                             
2022-09-27T16:19:32.3122537Z 
2022-09-27T16:19:32.3133146Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [  7%]
2022-09-27T16:19:32.3141207Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%]
2022-09-27T16:19:32.3148921Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%]
2022-09-27T16:19:32.3156460Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-09-27T16:19:32.3163766Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%]
2022-09-27T16:19:32.3170830Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%]
2022-09-27T16:19:32.3178049Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%]
2022-09-27T16:19:32.3185538Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-09-27T16:19:32.3193003Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%]
2022-09-27T16:19:32.3200736Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%]
2022-09-27T16:19:32.3209060Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%]
2022-09-27T16:19:32.3216008Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-09-27T16:19:32.3643332Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-09-27T16:19:32.3644098Z 
2022-09-27T16:19:32.3644452Z ======================== 1 passed, 12 skipped in 0.08s =========================
2022-09-27T16:19:32.5673193Z Running distributed/pipeline/sync/skip/test_leak ... [2022-09-27 16:19:32.566944]
2022-09-27T16:19:32.5673763Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-09-27 16:19:32.567022]
2022-09-27T16:19:33.9412600Z ============================= test session starts ==============================
2022-09-27T16:19:33.9413035Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:33.9484478Z cachedir: .pytest_cache
2022-09-27T16:19:33.9485399Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:33.9486288Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:33.9486724Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:33.9487176Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:33.9614256Z collecting ... 
2022-09-27T16:19:33.9614791Z collected 8 items                                                              
2022-09-27T16:19:33.9617339Z 
2022-09-27T16:19:34.0232688Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2022-09-27T16:19:34.0419495Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2022-09-27T16:19:34.0640782Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2022-09-27T16:19:34.0832578Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2022-09-27T16:19:34.1040502Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2022-09-27T16:19:34.1200821Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2022-09-27T16:19:34.1338692Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2022-09-27T16:19:34.1524540Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2022-09-27T16:19:34.1525544Z 
2022-09-27T16:19:34.1525901Z ============================== 8 passed in 0.21s ===============================
2022-09-27T16:19:34.3609972Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-09-27 16:19:34.360642]
2022-09-27T16:19:34.3610510Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-09-27 16:19:34.360723]
2022-09-27T16:19:35.7504517Z ============================= test session starts ==============================
2022-09-27T16:19:35.7504972Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:35.7574412Z cachedir: .pytest_cache
2022-09-27T16:19:35.7575036Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:35.7575381Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:35.7575632Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:35.7576024Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:35.7696829Z collecting ... 
2022-09-27T16:19:35.7697175Z collected 7 items                                                              
2022-09-27T16:19:35.7700756Z 
2022-09-27T16:19:35.7731330Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-09-27T16:19:35.7748297Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-09-27T16:19:35.7763696Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-09-27T16:19:35.7777471Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-09-27T16:19:35.7792250Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-09-27T16:19:35.7805795Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-09-27T16:19:35.7821638Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-09-27T16:19:35.7822188Z 
2022-09-27T16:19:35.7822560Z ============================== 7 passed in 0.03s ===============================
2022-09-27T16:19:35.9809677Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-09-27 16:19:35.980500]
2022-09-27T16:19:35.9810457Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-09-27 16:19:35.980581]
2022-09-27T16:19:37.3301813Z ============================= test session starts ==============================
2022-09-27T16:19:37.3302252Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:37.3372414Z cachedir: .pytest_cache
2022-09-27T16:19:37.3373039Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:37.3373433Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:37.3373676Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:37.3374072Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:37.3516718Z collecting ... 
2022-09-27T16:19:37.3517067Z collected 9 items                                                              
2022-09-27T16:19:37.3520070Z 
2022-09-27T16:19:37.3545793Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-09-27T16:19:37.3559797Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-09-27T16:19:37.3571762Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-09-27T16:19:37.3584956Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-09-27T16:19:37.3597806Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-09-27T16:19:37.3611836Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-09-27T16:19:37.3623549Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-09-27T16:19:37.3636978Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-09-27T16:19:37.3652350Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-09-27T16:19:37.3653483Z 
2022-09-27T16:19:37.3653767Z ============================== 9 passed in 0.04s ===============================
2022-09-27T16:19:37.5713148Z Running distributed/pipeline/sync/test_bugs ... [2022-09-27 16:19:37.570977]
2022-09-27T16:19:37.5713633Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-09-27 16:19:37.571058]
2022-09-27T16:19:38.9626820Z ============================= test session starts ==============================
2022-09-27T16:19:38.9627274Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:38.9697980Z cachedir: .pytest_cache
2022-09-27T16:19:38.9698608Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:38.9698958Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:38.9699200Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:38.9699609Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:38.9821758Z collecting ... 
2022-09-27T16:19:38.9822071Z collected 4 items                                                              
2022-09-27T16:19:38.9825531Z 
2022-09-27T16:19:39.0196013Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2022-09-27T16:19:39.0363820Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2022-09-27T16:19:39.0372498Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%]
2022-09-27T16:19:39.1447222Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2022-09-27T16:19:39.1447741Z 
2022-09-27T16:19:39.1448209Z ========================= 3 passed, 1 skipped in 0.18s =========================
2022-09-27T16:19:39.3528742Z Running distributed/pipeline/sync/test_copy ... [2022-09-27 16:19:39.352473]
2022-09-27T16:19:39.3529254Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-09-27 16:19:39.352551]
2022-09-27T16:19:40.6912141Z ============================= test session starts ==============================
2022-09-27T16:19:40.6912580Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:40.6983272Z cachedir: .pytest_cache
2022-09-27T16:19:40.6984140Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:40.6984561Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:40.6984817Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:40.6985225Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:40.7101071Z collecting ... 
2022-09-27T16:19:40.7101400Z collected 5 items                                                              
2022-09-27T16:19:40.7104854Z 
2022-09-27T16:19:40.7141073Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-09-27T16:19:40.7150430Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED  [ 40%]
2022-09-27T16:19:40.7158614Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED  [ 60%]
2022-09-27T16:19:40.7168128Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%]
2022-09-27T16:19:40.7184223Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-09-27T16:19:40.7184862Z 
2022-09-27T16:19:40.7185265Z ========================= 2 passed, 3 skipped in 0.03s =========================
2022-09-27T16:19:40.9205871Z Running distributed/pipeline/sync/test_dependency ... [2022-09-27 16:19:40.920175]
2022-09-27T16:19:40.9206384Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-09-27 16:19:40.920256]
2022-09-27T16:19:42.2627550Z ============================= test session starts ==============================
2022-09-27T16:19:42.2627969Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:42.2701690Z cachedir: .pytest_cache
2022-09-27T16:19:42.2702502Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:42.2703063Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:42.2703296Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:42.2703706Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:42.2868393Z collecting ... 
2022-09-27T16:19:42.2868968Z collected 6 items                                                              
2022-09-27T16:19:42.2871560Z 
2022-09-27T16:19:42.2881210Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED     [ 16%]
2022-09-27T16:19:42.2909835Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-09-27T16:19:42.2922700Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-09-27T16:19:42.2938928Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-09-27T16:19:42.2950943Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-09-27T16:19:42.2964781Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-09-27T16:19:42.2965659Z 
2022-09-27T16:19:42.2966085Z ========================= 5 passed, 1 skipped in 0.03s =========================
2022-09-27T16:19:42.5014912Z Running distributed/pipeline/sync/test_microbatch ... [2022-09-27 16:19:42.501131]
2022-09-27T16:19:42.5015487Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-09-27 16:19:42.501210]
2022-09-27T16:19:43.8720748Z ============================= test session starts ==============================
2022-09-27T16:19:43.8721230Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:43.8792514Z cachedir: .pytest_cache
2022-09-27T16:19:43.8793482Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:43.8794095Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:43.8794524Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:43.8794974Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:43.9013518Z collecting ... 
2022-09-27T16:19:43.9014103Z collected 10 items                                                             
2022-09-27T16:19:43.9016988Z 
2022-09-27T16:19:43.9040997Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2022-09-27T16:19:43.9053844Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2022-09-27T16:19:43.9065474Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2022-09-27T16:19:43.9077042Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2022-09-27T16:19:43.9088174Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2022-09-27T16:19:43.9102104Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2022-09-27T16:19:43.9116827Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2022-09-27T16:19:43.9128074Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2022-09-27T16:19:43.9139582Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2022-09-27T16:19:43.9153604Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2022-09-27T16:19:43.9154506Z 
2022-09-27T16:19:43.9154904Z ============================== 10 passed in 0.04s ==============================
2022-09-27T16:19:44.1176083Z Running distributed/pipeline/sync/test_pipe ... [2022-09-27 16:19:44.117216]
2022-09-27T16:19:44.1176579Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-09-27 16:19:44.117294]
2022-09-27T16:19:45.4898523Z ============================= test session starts ==============================
2022-09-27T16:19:45.4898972Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:45.4969491Z cachedir: .pytest_cache
2022-09-27T16:19:45.4970369Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:45.4970922Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:45.4971331Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:45.4972281Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:45.6066034Z collecting ... 
2022-09-27T16:19:45.6066655Z collected 56 items                                                             
2022-09-27T16:19:45.6069255Z 
2022-09-27T16:19:45.6098784Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-09-27T16:19:45.6440582Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-09-27T16:19:45.6602122Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-09-27T16:19:45.6761090Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-09-27T16:19:45.6933931Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-09-27T16:19:45.7160986Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-09-27T16:19:45.7359921Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-09-27T16:19:45.7542460Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-09-27T16:19:45.7720109Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-09-27T16:19:45.7882304Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-09-27T16:19:45.8032369Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-09-27T16:19:45.8203445Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-09-27T16:19:45.8347476Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-09-27T16:19:45.8560272Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-09-27T16:19:46.0760740Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-09-27T16:19:46.0939144Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-09-27T16:19:46.1159595Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-09-27T16:19:46.1359828Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-09-27T16:19:46.1559674Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-09-27T16:19:46.1739783Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-09-27T16:19:46.1920391Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-09-27T16:19:46.2099036Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-09-27T16:19:46.2322143Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-09-27T16:19:46.2562698Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-09-27T16:19:46.2802335Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-09-27T16:19:46.3040400Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-09-27T16:19:46.3202283Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-09-27T16:19:46.3344161Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-09-27T16:19:46.3493371Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-09-27T16:19:46.3720320Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-09-27T16:19:46.3950911Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-09-27T16:19:46.4122450Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-09-27T16:19:46.4281741Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-09-27T16:19:46.4458035Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-09-27T16:19:46.4724519Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-09-27T16:19:46.4922307Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-09-27T16:19:46.5121211Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-09-27T16:19:46.5279486Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-09-27T16:19:46.5481986Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-09-27T16:19:46.5641670Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-09-27T16:19:46.5839789Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-09-27T16:19:46.5847847Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED   [ 75%]
2022-09-27T16:19:46.6000334Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-09-27T16:19:46.6147741Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-09-27T16:19:46.6320629Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-09-27T16:19:46.6519426Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-09-27T16:19:46.6707657Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-09-27T16:19:46.6714850Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%]
2022-09-27T16:19:46.6721693Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%]
2022-09-27T16:19:46.6881459Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-09-27T16:19:47.0124022Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-09-27T16:19:47.0131425Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%]
2022-09-27T16:19:47.0138264Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%]
2022-09-27T16:19:47.0145375Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%]
2022-09-27T16:19:47.0152411Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%]
2022-09-27T16:19:47.0162992Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%]
2022-09-27T16:19:47.0164277Z 
2022-09-27T16:19:47.0164611Z ======================== 48 passed, 8 skipped in 1.53s =========================
2022-09-27T16:19:47.2267500Z Running distributed/pipeline/sync/test_stream ... [2022-09-27 16:19:47.226366]
2022-09-27T16:19:47.2268000Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-09-27 16:19:47.226468]
2022-09-27T16:19:48.6123673Z ============================= test session starts ==============================
2022-09-27T16:19:48.6124324Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:48.6194498Z cachedir: .pytest_cache
2022-09-27T16:19:48.6195338Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:48.6195915Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:48.6196154Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:48.6196576Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:48.6424460Z collecting ... 
2022-09-27T16:19:48.6424771Z collected 19 items                                                             
2022-09-27T16:19:48.6428044Z 
2022-09-27T16:19:48.6450468Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-09-27T16:19:48.6459006Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%]
2022-09-27T16:19:48.6471053Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-09-27T16:19:48.6478915Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%]
2022-09-27T16:19:48.6490394Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-09-27T16:19:48.6498122Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%]
2022-09-27T16:19:48.6511447Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-09-27T16:19:48.6519192Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%]
2022-09-27T16:19:48.6530283Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-09-27T16:19:48.6538009Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%]
2022-09-27T16:19:48.6549511Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-09-27T16:19:48.6557224Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%]
2022-09-27T16:19:48.6574608Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-09-27T16:19:48.6582357Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%]
2022-09-27T16:19:48.6590275Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%]
2022-09-27T16:19:48.6597949Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%]
2022-09-27T16:19:48.6610183Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-09-27T16:19:48.6617817Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%]
2022-09-27T16:19:48.6628698Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%]
2022-09-27T16:19:48.6630178Z 
2022-09-27T16:19:48.6630442Z ======================== 8 passed, 11 skipped in 0.05s =========================
2022-09-27T16:19:48.8669892Z Running distributed/pipeline/sync/test_worker ... [2022-09-27 16:19:48.866641]
2022-09-27T16:19:48.8670388Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-09-27 16:19:48.866724]
2022-09-27T16:19:50.2119042Z ============================= test session starts ==============================
2022-09-27T16:19:50.2119700Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:19:50.2190251Z cachedir: .pytest_cache
2022-09-27T16:19:50.2190930Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:19:50.2191452Z torch: 1.13.0a0+git52424e2
2022-09-27T16:19:50.2191704Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:19:50.2192106Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:19:50.2341013Z collecting ... 
2022-09-27T16:19:50.2341406Z collected 6 items                                                              
2022-09-27T16:19:50.2344726Z 
2022-09-27T16:19:50.2374738Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2022-09-27T16:19:50.2390516Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2022-09-27T16:19:50.2404048Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2022-09-27T16:19:50.2421752Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2022-09-27T16:19:50.2436250Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2022-09-27T16:19:50.2454234Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2022-09-27T16:19:50.2455679Z 
2022-09-27T16:19:50.2455948Z ============================== 6 passed in 0.03s ===============================
2022-09-27T16:19:50.7113787Z 
2022-09-27T16:19:50.7114268Z real	51m13.060s
2022-09-27T16:19:50.7114667Z user	92m7.375s
2022-09-27T16:19:50.7114913Z sys	24m26.257s
2022-09-27T16:19:50.7115081Z + assert_git_not_dirty
2022-09-27T16:19:50.7115518Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2022-09-27T16:19:50.7115820Z + [[ linux-focal-py3.7-gcc7 != *xla* ]]
2022-09-27T16:19:50.7116347Z ++ git status --porcelain
2022-09-27T16:19:51.6670528Z + git_status=
2022-09-27T16:19:51.6670895Z + [[ -n '' ]]
2022-09-27T16:19:51.6671220Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-09-27T16:19:51.6671427Z + [[ 2 == 1 ]]
2022-09-27T16:19:51.6737059Z Prepare all required actions
2022-09-27T16:19:51.6737365Z Getting action download info
2022-09-27T16:19:51.8822138Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-09-27T16:19:51.8822364Z with:
2022-09-27T16:19:51.8822721Z   github-token: ***
2022-09-27T16:19:51.8822882Z env:
2022-09-27T16:19:51.8823064Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:51.8823252Z ##[endgroup]
2022-09-27T16:19:51.8851639Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-09-27T16:19:51.8851877Z with:
2022-09-27T16:19:51.8852029Z   shell: bash
2022-09-27T16:19:51.8852210Z   timeout_minutes: 10
2022-09-27T16:19:51.8852389Z   max_attempts: 5
2022-09-27T16:19:51.8852558Z   retry_wait_seconds: 30
2022-09-27T16:19:51.8852946Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-09-27T16:19:51.8853385Z   polling_interval_seconds: 1
2022-09-27T16:19:51.8853571Z   warning_on_retry: true
2022-09-27T16:19:51.8853762Z   continue_on_error: false
2022-09-27T16:19:51.8853939Z env:
2022-09-27T16:19:51.8854097Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:51.8854409Z   GITHUB_TOKEN: ***
2022-09-27T16:19:51.8854592Z ##[endgroup]
2022-09-27T16:19:51.9238565Z 
2022-09-27T16:19:51.9284179Z + python3 -m pip install requests==2.26.0
2022-09-27T16:19:52.1345476Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:19:52.1521130Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-09-27T16:19:52.1647732Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-09-27T16:19:52.1667236Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12)
2022-09-27T16:19:52.1819279Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-09-27T16:19:52.1830419Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-09-27T16:19:52.3338625Z ++ python3 .github/scripts/get_workflow_job_id.py 3133193930 i-01f41056b46f41785
2022-09-27T16:19:55.9540386Z + GHA_WORKFLOW_JOB_ID=8576161444
2022-09-27T16:19:55.9540934Z + echo '::set-output name=job-id::8576161444'
2022-09-27T16:19:56.9334542Z Command completed after 1 attempt(s).
2022-09-27T16:19:56.9334803Z 
2022-09-27T16:19:56.9438731Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:19:56.9438958Z kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:19:56.9450628Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:19:56.9450859Z env:
2022-09-27T16:19:56.9451039Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:56.9451227Z   MONITOR_SCRIPT_PID: 6041
2022-09-27T16:19:56.9451414Z ##[endgroup]
2022-09-27T16:19:56.9539488Z Prepare all required actions
2022-09-27T16:19:56.9539744Z Getting action download info
2022-09-27T16:19:57.1449985Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-09-27T16:19:57.2933504Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-09-27T16:19:57.2933728Z with:
2022-09-27T16:19:57.2933954Z   file-suffix: test-distributed-2-2-linux.2xlarge_8576161444
2022-09-27T16:19:57.2934188Z env:
2022-09-27T16:19:57.2934364Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.2934539Z ##[endgroup]
2022-09-27T16:19:57.2957703Z ##[group]Run # Remove any previous test jsons if they exist
2022-09-27T16:19:57.2957983Z # Remove any previous test jsons if they exist
2022-09-27T16:19:57.2958214Z rm -f test-jsons-*.zip
2022-09-27T16:19:57.2958451Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-09-27T16:19:57.2969542Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:19:57.2969789Z env:
2022-09-27T16:19:57.2969952Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.2970207Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_8576161444
2022-09-27T16:19:57.2970442Z ##[endgroup]
2022-09-27T16:19:57.3096720Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-09-27T16:19:57.3123372Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-09-27T16:19:57.3129193Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-09-27T16:19:57.3130131Z   adding: test/.pytorch-slow-tests.json (deflated 75%)
2022-09-27T16:19:57.3135464Z   adding: test/.pytorch-disabled-tests.json (deflated 85%)
2022-09-27T16:19:57.3154689Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:19:57.3154981Z # Remove any previous test reports if they exist
2022-09-27T16:19:57.3155229Z rm -f test-reports-*.zip
2022-09-27T16:19:57.3155473Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-09-27T16:19:57.3165769Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:19:57.3165995Z env:
2022-09-27T16:19:57.3166159Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.3166420Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_8576161444
2022-09-27T16:19:57.3166659Z ##[endgroup]
2022-09-27T16:19:57.3288766Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152841.xml (deflated 38%)
2022-09-27T16:19:57.3289289Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152844.xml (deflated 40%)
2022-09-27T16:19:57.3289811Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152847.xml (deflated 38%)
2022-09-27T16:19:57.3290318Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152851.xml (deflated 39%)
2022-09-27T16:19:57.3290926Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152854.xml (deflated 39%)
2022-09-27T16:19:57.3291418Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152857.xml (deflated 40%)
2022-09-27T16:19:57.3291924Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152900.xml (deflated 39%)
2022-09-27T16:19:57.3292431Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152904.xml (deflated 39%)
2022-09-27T16:19:57.3292931Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152907.xml (deflated 39%)
2022-09-27T16:19:57.3293427Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152910.xml (deflated 39%)
2022-09-27T16:19:57.3293928Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927152913.xml (deflated 40%)
2022-09-27T16:19:57.3294503Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152917.xml (deflated 38%)
2022-09-27T16:19:57.3295034Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152920.xml (deflated 40%)
2022-09-27T16:19:57.3295545Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152923.xml (deflated 39%)
2022-09-27T16:19:57.3296072Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152926.xml (deflated 40%)
2022-09-27T16:19:57.3296595Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152930.xml (deflated 39%)
2022-09-27T16:19:57.3297114Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152933.xml (deflated 40%)
2022-09-27T16:19:57.3297612Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152936.xml (deflated 39%)
2022-09-27T16:19:57.3298150Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152939.xml (deflated 40%)
2022-09-27T16:19:57.3298666Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152942.xml (deflated 38%)
2022-09-27T16:19:57.3299181Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152946.xml (deflated 38%)
2022-09-27T16:19:57.3299682Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927152949.xml (deflated 40%)
2022-09-27T16:19:57.3300251Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152952.xml (deflated 44%)
2022-09-27T16:19:57.3300871Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152955.xml (deflated 44%)
2022-09-27T16:19:57.3301485Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927152959.xml (deflated 42%)
2022-09-27T16:19:57.3302114Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153002.xml (deflated 42%)
2022-09-27T16:19:57.3302711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153005.xml (deflated 45%)
2022-09-27T16:19:57.3303315Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153008.xml (deflated 45%)
2022-09-27T16:19:57.3304115Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153011.xml (deflated 46%)
2022-09-27T16:19:57.3304811Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153015.xml (deflated 46%)
2022-09-27T16:19:57.3305476Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153018.xml (deflated 44%)
2022-09-27T16:19:57.3306083Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153021.xml (deflated 45%)
2022-09-27T16:19:57.3306695Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153024.xml (deflated 45%)
2022-09-27T16:19:57.3307305Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153028.xml (deflated 44%)
2022-09-27T16:19:57.3307907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153031.xml (deflated 43%)
2022-09-27T16:19:57.3308515Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153034.xml (deflated 43%)
2022-09-27T16:19:57.3309119Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153037.xml (deflated 43%)
2022-09-27T16:19:57.3309772Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153041.xml (deflated 45%)
2022-09-27T16:19:57.3310371Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153044.xml (deflated 44%)
2022-09-27T16:19:57.3310978Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153047.xml (deflated 45%)
2022-09-27T16:19:57.3311578Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153050.xml (deflated 45%)
2022-09-27T16:19:57.3312181Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153054.xml (deflated 50%)
2022-09-27T16:19:57.3312776Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153057.xml (deflated 42%)
2022-09-27T16:19:57.3313391Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153100.xml (deflated 42%)
2022-09-27T16:19:57.3314122Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153103.xml (deflated 42%)
2022-09-27T16:19:57.3314833Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153106.xml (deflated 42%)
2022-09-27T16:19:57.3315508Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153110.xml (deflated 43%)
2022-09-27T16:19:57.3316152Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153113.xml (deflated 42%)
2022-09-27T16:19:57.3316828Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153116.xml (deflated 42%)
2022-09-27T16:19:57.3317511Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153119.xml (deflated 42%)
2022-09-27T16:19:57.3318188Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153122.xml (deflated 41%)
2022-09-27T16:19:57.3318848Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153126.xml (deflated 45%)
2022-09-27T16:19:57.3319525Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153129.xml (deflated 46%)
2022-09-27T16:19:57.3320206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153132.xml (deflated 41%)
2022-09-27T16:19:57.3320864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153136.xml (deflated 42%)
2022-09-27T16:19:57.3321563Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153139.xml (deflated 42%)
2022-09-27T16:19:57.3322235Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153142.xml (deflated 42%)
2022-09-27T16:19:57.3322907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153145.xml (deflated 42%)
2022-09-27T16:19:57.3323582Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927153149.xml (deflated 41%)
2022-09-27T16:19:57.3324236Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153152.xml (deflated 39%)
2022-09-27T16:19:57.3324929Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153155.xml (deflated 41%)
2022-09-27T16:19:57.3325548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153159.xml (deflated 39%)
2022-09-27T16:19:57.3326116Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153202.xml (deflated 40%)
2022-09-27T16:19:57.3326670Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153206.xml (deflated 39%)
2022-09-27T16:19:57.3327234Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153209.xml (deflated 40%)
2022-09-27T16:19:57.3327807Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153213.xml (deflated 39%)
2022-09-27T16:19:57.3328370Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153218.xml (deflated 41%)
2022-09-27T16:19:57.3328970Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153221.xml (deflated 40%)
2022-09-27T16:19:57.3329524Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153225.xml (deflated 41%)
2022-09-27T16:19:57.3330091Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153228.xml (deflated 40%)
2022-09-27T16:19:57.3330662Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153232.xml (deflated 39%)
2022-09-27T16:19:57.3331226Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153235.xml (deflated 40%)
2022-09-27T16:19:57.3331776Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153239.xml (deflated 40%)
2022-09-27T16:19:57.3332340Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153242.xml (deflated 40%)
2022-09-27T16:19:57.3332910Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153245.xml (deflated 40%)
2022-09-27T16:19:57.3333476Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153249.xml (deflated 41%)
2022-09-27T16:19:57.3334026Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153252.xml (deflated 40%)
2022-09-27T16:19:57.3334588Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153257.xml (deflated 40%)
2022-09-27T16:19:57.3335152Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153301.xml (deflated 41%)
2022-09-27T16:19:57.3335717Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153304.xml (deflated 40%)
2022-09-27T16:19:57.3336300Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153308.xml (deflated 40%)
2022-09-27T16:19:57.3336864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153311.xml (deflated 41%)
2022-09-27T16:19:57.3337425Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153315.xml (deflated 40%)
2022-09-27T16:19:57.3337988Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153318.xml (deflated 40%)
2022-09-27T16:19:57.3338537Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153322.xml (deflated 41%)
2022-09-27T16:19:57.3339098Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153325.xml (deflated 40%)
2022-09-27T16:19:57.3339656Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153329.xml (deflated 40%)
2022-09-27T16:19:57.3340252Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153332.xml (deflated 41%)
2022-09-27T16:19:57.3340801Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153336.xml (deflated 39%)
2022-09-27T16:19:57.3341361Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153339.xml (deflated 39%)
2022-09-27T16:19:57.3341921Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153343.xml (deflated 39%)
2022-09-27T16:19:57.3342485Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153348.xml (deflated 41%)
2022-09-27T16:19:57.3343033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153351.xml (deflated 39%)
2022-09-27T16:19:57.3343601Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153355.xml (deflated 40%)
2022-09-27T16:19:57.3344368Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153358.xml (deflated 41%)
2022-09-27T16:19:57.3344940Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153402.xml (deflated 40%)
2022-09-27T16:19:57.3345489Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153405.xml (deflated 39%)
2022-09-27T16:19:57.3346054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153410.xml (deflated 41%)
2022-09-27T16:19:57.3346626Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153413.xml (deflated 39%)
2022-09-27T16:19:57.3347196Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153417.xml (deflated 39%)
2022-09-27T16:19:57.3347748Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153421.xml (deflated 40%)
2022-09-27T16:19:57.3348311Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153424.xml (deflated 41%)
2022-09-27T16:19:57.3348874Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153427.xml (deflated 39%)
2022-09-27T16:19:57.3349463Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153431.xml (deflated 39%)
2022-09-27T16:19:57.3350023Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153436.xml (deflated 41%)
2022-09-27T16:19:57.3350572Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153438.xml (deflated 40%)
2022-09-27T16:19:57.3351206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153441.xml (deflated 41%)
2022-09-27T16:19:57.3351775Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153442.xml (deflated 41%)
2022-09-27T16:19:57.3352342Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927153446.xml (deflated 39%)
2022-09-27T16:19:57.3352870Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153449.xml (deflated 38%)
2022-09-27T16:19:57.3353390Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153451.xml (deflated 39%)
2022-09-27T16:19:57.3353913Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153452.xml (deflated 39%)
2022-09-27T16:19:57.3354438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153453.xml (deflated 39%)
2022-09-27T16:19:57.3354976Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153455.xml (deflated 38%)
2022-09-27T16:19:57.3355482Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927153456.xml (deflated 39%)
2022-09-27T16:19:57.3356020Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220927153458.xml (deflated 39%)
2022-09-27T16:19:57.3356553Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220927153459.xml (deflated 41%)
2022-09-27T16:19:57.3357154Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220927153501.xml (deflated 93%)
2022-09-27T16:19:57.3357850Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220927153501.xml (deflated 68%)
2022-09-27T16:19:57.3358515Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20220927153621.xml (deflated 73%)
2022-09-27T16:19:57.3359173Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220927160504.xml (deflated 42%)
2022-09-27T16:19:57.3359841Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220927160515.xml (deflated 42%)
2022-09-27T16:19:57.3360494Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160525.xml (deflated 41%)
2022-09-27T16:19:57.3361114Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160535.xml (deflated 41%)
2022-09-27T16:19:57.3361734Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160545.xml (deflated 41%)
2022-09-27T16:19:57.3362332Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160600.xml (deflated 42%)
2022-09-27T16:19:57.3362940Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160610.xml (deflated 41%)
2022-09-27T16:19:57.3363548Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160614.xml (deflated 42%)
2022-09-27T16:19:57.3364152Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160624.xml (deflated 41%)
2022-09-27T16:19:57.3364740Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160633.xml (deflated 40%)
2022-09-27T16:19:57.3365340Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160639.xml (deflated 41%)
2022-09-27T16:19:57.3365981Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160649.xml (deflated 41%)
2022-09-27T16:19:57.3366587Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160659.xml (deflated 41%)
2022-09-27T16:19:57.3367192Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160708.xml (deflated 41%)
2022-09-27T16:19:57.3367781Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160714.xml (deflated 41%)
2022-09-27T16:19:57.3368384Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160731.xml (deflated 41%)
2022-09-27T16:19:57.3368988Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160743.xml (deflated 41%)
2022-09-27T16:19:57.3369630Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160749.xml (deflated 41%)
2022-09-27T16:19:57.3370223Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160759.xml (deflated 41%)
2022-09-27T16:19:57.3370821Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160806.xml (deflated 41%)
2022-09-27T16:19:57.3371420Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160816.xml (deflated 41%)
2022-09-27T16:19:57.3372021Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220927160826.xml (deflated 40%)
2022-09-27T16:19:57.3372631Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160836.xml (deflated 41%)
2022-09-27T16:19:57.3373280Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160846.xml (deflated 41%)
2022-09-27T16:19:57.3373915Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160856.xml (deflated 42%)
2022-09-27T16:19:57.3374549Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160906.xml (deflated 41%)
2022-09-27T16:19:57.3375170Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160912.xml (deflated 41%)
2022-09-27T16:19:57.3375794Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220927160920.xml (deflated 42%)
2022-09-27T16:19:57.3376428Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220927160929.xml (deflated 94%)
2022-09-27T16:19:57.3377112Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220927160929.xml (deflated 87%)
2022-09-27T16:19:57.3377703Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220927161121.xml (deflated 81%)
2022-09-27T16:19:57.3378227Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220927161121.xml (deflated 64%)
2022-09-27T16:19:57.3378769Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220927161121.xml (deflated 62%)
2022-09-27T16:19:57.3379334Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220927161121.xml (deflated 93%)
2022-09-27T16:19:57.3379923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20220927161256.xml (deflated 86%)
2022-09-27T16:19:57.3380557Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20220927161256.xml (deflated 86%)
2022-09-27T16:19:57.3381244Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220927161351.xml (deflated 91%)
2022-09-27T16:19:57.3382008Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220927161351.xml (deflated 73%)
2022-09-27T16:19:57.3382695Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220927161431.xml (deflated 92%)
2022-09-27T16:19:57.3383269Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220927161458.xml (deflated 93%)
2022-09-27T16:19:57.3383992Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161522.xml (deflated 41%)
2022-09-27T16:19:57.3384756Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161525.xml (deflated 40%)
2022-09-27T16:19:57.3385384Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161529.xml (deflated 41%)
2022-09-27T16:19:57.3385987Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161533.xml (deflated 41%)
2022-09-27T16:19:57.3386613Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161536.xml (deflated 40%)
2022-09-27T16:19:57.3387248Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161540.xml (deflated 41%)
2022-09-27T16:19:57.3387875Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161543.xml (deflated 41%)
2022-09-27T16:19:57.3388486Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161547.xml (deflated 41%)
2022-09-27T16:19:57.3389107Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927161550.xml (deflated 40%)
2022-09-27T16:19:57.3389735Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161554.xml (deflated 41%)
2022-09-27T16:19:57.3390355Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161555.xml (deflated 41%)
2022-09-27T16:19:57.3390973Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161557.xml (deflated 41%)
2022-09-27T16:19:57.3391576Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161558.xml (deflated 41%)
2022-09-27T16:19:57.3392205Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927161559.xml (deflated 41%)
2022-09-27T16:19:57.3392890Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161602.xml (deflated 44%)
2022-09-27T16:19:57.3393615Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161604.xml (deflated 44%)
2022-09-27T16:19:57.3394315Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927161606.xml (deflated 44%)
2022-09-27T16:19:57.3394996Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161607.xml (deflated 41%)
2022-09-27T16:19:57.3395682Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161609.xml (deflated 42%)
2022-09-27T16:19:57.3396319Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161611.xml (deflated 42%)
2022-09-27T16:19:57.3396941Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220927161612.xml (deflated 42%)
2022-09-27T16:19:57.3397581Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161614.xml (deflated 43%)
2022-09-27T16:19:57.3398232Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161617.xml (deflated 43%)
2022-09-27T16:19:57.3398883Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161621.xml (deflated 43%)
2022-09-27T16:19:57.3399553Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161624.xml (deflated 43%)
2022-09-27T16:19:57.3400199Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161628.xml (deflated 43%)
2022-09-27T16:19:57.3400853Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161631.xml (deflated 43%)
2022-09-27T16:19:57.3401501Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161635.xml (deflated 43%)
2022-09-27T16:19:57.3402133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927161638.xml (deflated 43%)
2022-09-27T16:19:57.3402741Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220927161642.xml (deflated 91%)
2022-09-27T16:19:57.3403352Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220927161657.xml (deflated 68%)
2022-09-27T16:19:57.3404038Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220927161657.xml (deflated 85%)
2022-09-27T16:19:57.3404701Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220927161713.xml (deflated 81%)
2022-09-27T16:19:57.3405298Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220927161727.xml (deflated 87%)
2022-09-27T16:19:57.3405929Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220927161739.xml (deflated 77%)
2022-09-27T16:19:57.3406554Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220927161816.xml (deflated 69%)
2022-09-27T16:19:57.3407130Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220927161824.xml (deflated 64%)
2022-09-27T16:19:57.3407778Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220927161830.xml (deflated 78%)
2022-09-27T16:19:57.3408437Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220927161830.xml (deflated 61%)
2022-09-27T16:19:57.3409060Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220927161835.xml (deflated 59%)
2022-09-27T16:19:57.3409609Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220927161840.xml (deflated 59%)
2022-09-27T16:19:57.3410158Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220927161846.xml (deflated 38%)
2022-09-27T16:19:57.3410768Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220927161849.xml (deflated 42%)
2022-09-27T16:19:57.3411385Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220927161853.xml (deflated 46%)
2022-09-27T16:19:57.3411978Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220927161856.xml (deflated 42%)
2022-09-27T16:19:57.3412572Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220927161900.xml (deflated 81%)
2022-09-27T16:19:57.3413182Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220927161900.xml (deflated 85%)
2022-09-27T16:19:57.3413826Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220927161900.xml (deflated 85%)
2022-09-27T16:19:57.3414468Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220927161901.xml (deflated 63%)
2022-09-27T16:19:57.3415036Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220927161901.xml (deflated 69%)
2022-09-27T16:19:57.3415699Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220927161903.xml (deflated 42%)
2022-09-27T16:19:57.3416415Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220927161903.xml (deflated 74%)
2022-09-27T16:19:57.3417209Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927161903.xml (deflated 45%)
2022-09-27T16:19:57.3417913Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220927161905.xml (deflated 55%)
2022-09-27T16:19:57.3418465Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220927161907.xml (deflated 68%)
2022-09-27T16:19:57.3419010Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220927161908.xml (deflated 64%)
2022-09-27T16:19:57.3419599Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220927161910.xml (deflated 63%)
2022-09-27T16:19:57.3420172Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220927161912.xml (deflated 90%)
2022-09-27T16:19:57.3420778Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220927161912.xml (deflated 91%)
2022-09-27T16:19:57.3421438Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220927161913.xml (deflated 88%)
2022-09-27T16:19:57.3422088Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20220927161915.xml (deflated 77%)
2022-09-27T16:19:57.3422740Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220927161916.xml (deflated 70%)
2022-09-27T16:19:57.3423377Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220927161917.xml (deflated 78%)
2022-09-27T16:19:57.3424111Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220927161919.xml (deflated 69%)
2022-09-27T16:19:57.3424771Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220927161919.xml (deflated 62%)
2022-09-27T16:19:57.3425545Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220927161920.xml (deflated 82%)
2022-09-27T16:19:57.3426241Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220927161921.xml (deflated 61%)
2022-09-27T16:19:57.3426901Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220927161923.xml (deflated 62%)
2022-09-27T16:19:57.3427587Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220927161924.xml (deflated 44%)
2022-09-27T16:19:57.3428290Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220927161926.xml (deflated 75%)
2022-09-27T16:19:57.3428930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153624.xml (deflated 42%)
2022-09-27T16:19:57.3429574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153628.xml (deflated 42%)
2022-09-27T16:19:57.3430165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153629.xml (deflated 42%)
2022-09-27T16:19:57.3430762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153633.xml (deflated 43%)
2022-09-27T16:19:57.3431364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153637.xml (deflated 41%)
2022-09-27T16:19:57.3431956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153641.xml (deflated 41%)
2022-09-27T16:19:57.3432540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153644.xml (deflated 42%)
2022-09-27T16:19:57.3433142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153648.xml (deflated 41%)
2022-09-27T16:19:57.3433739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153652.xml (deflated 41%)
2022-09-27T16:19:57.3434339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153656.xml (deflated 41%)
2022-09-27T16:19:57.3434923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153700.xml (deflated 40%)
2022-09-27T16:19:57.3435605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153701.xml (deflated 41%)
2022-09-27T16:19:57.3436204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153705.xml (deflated 41%)
2022-09-27T16:19:57.3436800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153709.xml (deflated 42%)
2022-09-27T16:19:57.3437379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153710.xml (deflated 41%)
2022-09-27T16:19:57.3437969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153714.xml (deflated 42%)
2022-09-27T16:19:57.3438558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153718.xml (deflated 43%)
2022-09-27T16:19:57.3439148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153722.xml (deflated 42%)
2022-09-27T16:19:57.3439724Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153723.xml (deflated 46%)
2022-09-27T16:19:57.3440356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153727.xml (deflated 47%)
2022-09-27T16:19:57.3440946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153731.xml (deflated 48%)
2022-09-27T16:19:57.3441532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153734.xml (deflated 46%)
2022-09-27T16:19:57.3442123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153738.xml (deflated 40%)
2022-09-27T16:19:57.3442696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153742.xml (deflated 40%)
2022-09-27T16:19:57.3443280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153746.xml (deflated 40%)
2022-09-27T16:19:57.3443870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153750.xml (deflated 41%)
2022-09-27T16:19:57.3444490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153754.xml (deflated 40%)
2022-09-27T16:19:57.3445067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153757.xml (deflated 41%)
2022-09-27T16:19:57.3445662Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153801.xml (deflated 40%)
2022-09-27T16:19:57.3446376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153805.xml (deflated 42%)
2022-09-27T16:19:57.3446968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153807.xml (deflated 41%)
2022-09-27T16:19:57.3447545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153808.xml (deflated 40%)
2022-09-27T16:19:57.3448141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153812.xml (deflated 40%)
2022-09-27T16:19:57.3448735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153816.xml (deflated 43%)
2022-09-27T16:19:57.3449327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153818.xml (deflated 42%)
2022-09-27T16:19:57.3449910Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153819.xml (deflated 40%)
2022-09-27T16:19:57.3450500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153823.xml (deflated 40%)
2022-09-27T16:19:57.3451144Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153827.xml (deflated 43%)
2022-09-27T16:19:57.3451739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153828.xml (deflated 41%)
2022-09-27T16:19:57.3452330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153832.xml (deflated 40%)
2022-09-27T16:19:57.3452906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153836.xml (deflated 41%)
2022-09-27T16:19:57.3453498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153840.xml (deflated 40%)
2022-09-27T16:19:57.3454094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153844.xml (deflated 40%)
2022-09-27T16:19:57.3454686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153848.xml (deflated 40%)
2022-09-27T16:19:57.3455312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153852.xml (deflated 41%)
2022-09-27T16:19:57.3455904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153856.xml (deflated 40%)
2022-09-27T16:19:57.3456491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153900.xml (deflated 41%)
2022-09-27T16:19:57.3457084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153903.xml (deflated 41%)
2022-09-27T16:19:57.3457659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153907.xml (deflated 41%)
2022-09-27T16:19:57.3458251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153911.xml (deflated 41%)
2022-09-27T16:19:57.3458880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153915.xml (deflated 41%)
2022-09-27T16:19:57.3459470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153919.xml (deflated 41%)
2022-09-27T16:19:57.3460044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153922.xml (deflated 41%)
2022-09-27T16:19:57.3460631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153926.xml (deflated 41%)
2022-09-27T16:19:57.3461218Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153930.xml (deflated 41%)
2022-09-27T16:19:57.3461802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153934.xml (deflated 40%)
2022-09-27T16:19:57.3462379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153938.xml (deflated 40%)
2022-09-27T16:19:57.3462962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153942.xml (deflated 40%)
2022-09-27T16:19:57.3463548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153945.xml (deflated 41%)
2022-09-27T16:19:57.3464402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153949.xml (deflated 40%)
2022-09-27T16:19:57.3464987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153953.xml (deflated 41%)
2022-09-27T16:19:57.3465574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927153957.xml (deflated 41%)
2022-09-27T16:19:57.3466169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154001.xml (deflated 42%)
2022-09-27T16:19:57.3466763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154005.xml (deflated 41%)
2022-09-27T16:19:57.3467349Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154008.xml (deflated 41%)
2022-09-27T16:19:57.3467923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154012.xml (deflated 42%)
2022-09-27T16:19:57.3468508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154016.xml (deflated 41%)
2022-09-27T16:19:57.3469093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154020.xml (deflated 41%)
2022-09-27T16:19:57.3469677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154024.xml (deflated 41%)
2022-09-27T16:19:57.3470322Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154028.xml (deflated 42%)
2022-09-27T16:19:57.3470914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154032.xml (deflated 41%)
2022-09-27T16:19:57.3471504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154035.xml (deflated 41%)
2022-09-27T16:19:57.3472095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154039.xml (deflated 43%)
2022-09-27T16:19:57.3472675Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154041.xml (deflated 42%)
2022-09-27T16:19:57.3473269Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154042.xml (deflated 42%)
2022-09-27T16:19:57.3473907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154044.xml (deflated 42%)
2022-09-27T16:19:57.3474498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154045.xml (deflated 43%)
2022-09-27T16:19:57.3475075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154047.xml (deflated 42%)
2022-09-27T16:19:57.3475663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154048.xml (deflated 43%)
2022-09-27T16:19:57.3476252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154050.xml (deflated 42%)
2022-09-27T16:19:57.3476844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154051.xml (deflated 43%)
2022-09-27T16:19:57.3477426Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154053.xml (deflated 42%)
2022-09-27T16:19:57.3478016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154054.xml (deflated 43%)
2022-09-27T16:19:57.3478603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154056.xml (deflated 42%)
2022-09-27T16:19:57.3479192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154057.xml (deflated 42%)
2022-09-27T16:19:57.3479768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154059.xml (deflated 43%)
2022-09-27T16:19:57.3480356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154100.xml (deflated 43%)
2022-09-27T16:19:57.3480951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154102.xml (deflated 43%)
2022-09-27T16:19:57.3481543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154104.xml (deflated 43%)
2022-09-27T16:19:57.3482119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154105.xml (deflated 42%)
2022-09-27T16:19:57.3482709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154107.xml (deflated 43%)
2022-09-27T16:19:57.3483297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154108.xml (deflated 42%)
2022-09-27T16:19:57.3483886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154110.xml (deflated 42%)
2022-09-27T16:19:57.3484510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154111.xml (deflated 43%)
2022-09-27T16:19:57.3485099Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154113.xml (deflated 42%)
2022-09-27T16:19:57.3485687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154114.xml (deflated 42%)
2022-09-27T16:19:57.3486275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154116.xml (deflated 42%)
2022-09-27T16:19:57.3486852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154120.xml (deflated 42%)
2022-09-27T16:19:57.3487434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154121.xml (deflated 42%)
2022-09-27T16:19:57.3488025Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154123.xml (deflated 41%)
2022-09-27T16:19:57.3488643Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154128.xml (deflated 42%)
2022-09-27T16:19:57.3489218Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154132.xml (deflated 40%)
2022-09-27T16:19:57.3489807Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154137.xml (deflated 41%)
2022-09-27T16:19:57.3490394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154141.xml (deflated 41%)
2022-09-27T16:19:57.3490981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154145.xml (deflated 42%)
2022-09-27T16:19:57.3491553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154149.xml (deflated 40%)
2022-09-27T16:19:57.3492144Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154154.xml (deflated 42%)
2022-09-27T16:19:57.3492728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154155.xml (deflated 41%)
2022-09-27T16:19:57.3493318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154204.xml (deflated 40%)
2022-09-27T16:19:57.3493892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154208.xml (deflated 40%)
2022-09-27T16:19:57.3494475Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154212.xml (deflated 42%)
2022-09-27T16:19:57.3495062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154213.xml (deflated 41%)
2022-09-27T16:19:57.3495654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154215.xml (deflated 41%)
2022-09-27T16:19:57.3496228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154217.xml (deflated 41%)
2022-09-27T16:19:57.3496816Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154218.xml (deflated 41%)
2022-09-27T16:19:57.3497400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154220.xml (deflated 41%)
2022-09-27T16:19:57.3497988Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154221.xml (deflated 41%)
2022-09-27T16:19:57.3498556Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154223.xml (deflated 41%)
2022-09-27T16:19:57.3499180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154224.xml (deflated 41%)
2022-09-27T16:19:57.3499762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154228.xml (deflated 42%)
2022-09-27T16:19:57.3500344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154230.xml (deflated 40%)
2022-09-27T16:19:57.3500940Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154234.xml (deflated 40%)
2022-09-27T16:19:57.3501513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154238.xml (deflated 41%)
2022-09-27T16:19:57.3502098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154241.xml (deflated 42%)
2022-09-27T16:19:57.3502726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154243.xml (deflated 41%)
2022-09-27T16:19:57.3503314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154245.xml (deflated 41%)
2022-09-27T16:19:57.3504010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154246.xml (deflated 42%)
2022-09-27T16:19:57.3504693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154250.xml (deflated 41%)
2022-09-27T16:19:57.3505285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154254.xml (deflated 41%)
2022-09-27T16:19:57.3505873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154258.xml (deflated 41%)
2022-09-27T16:19:57.3506459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154301.xml (deflated 41%)
2022-09-27T16:19:57.3507050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154303.xml (deflated 41%)
2022-09-27T16:19:57.3507638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154307.xml (deflated 41%)
2022-09-27T16:19:57.3508225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154311.xml (deflated 41%)
2022-09-27T16:19:57.3508800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154312.xml (deflated 41%)
2022-09-27T16:19:57.3509381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154314.xml (deflated 40%)
2022-09-27T16:19:57.3509965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154317.xml (deflated 41%)
2022-09-27T16:19:57.3510560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154319.xml (deflated 41%)
2022-09-27T16:19:57.3511129Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154321.xml (deflated 42%)
2022-09-27T16:19:57.3511715Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154324.xml (deflated 41%)
2022-09-27T16:19:57.3512303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154328.xml (deflated 41%)
2022-09-27T16:19:57.3512887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154332.xml (deflated 41%)
2022-09-27T16:19:57.3513458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154335.xml (deflated 41%)
2022-09-27T16:19:57.3514118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154339.xml (deflated 41%)
2022-09-27T16:19:57.3514707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154343.xml (deflated 42%)
2022-09-27T16:19:57.3515299Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154347.xml (deflated 42%)
2022-09-27T16:19:57.3515875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154350.xml (deflated 42%)
2022-09-27T16:19:57.3516461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154354.xml (deflated 42%)
2022-09-27T16:19:57.3517050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154358.xml (deflated 42%)
2022-09-27T16:19:57.3517683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154401.xml (deflated 41%)
2022-09-27T16:19:57.3518351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154405.xml (deflated 42%)
2022-09-27T16:19:57.3518941Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154409.xml (deflated 42%)
2022-09-27T16:19:57.3519516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154413.xml (deflated 42%)
2022-09-27T16:19:57.3520102Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154416.xml (deflated 42%)
2022-09-27T16:19:57.3520689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154420.xml (deflated 42%)
2022-09-27T16:19:57.3521282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154424.xml (deflated 42%)
2022-09-27T16:19:57.3521852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154428.xml (deflated 41%)
2022-09-27T16:19:57.3522433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154429.xml (deflated 42%)
2022-09-27T16:19:57.3523020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154433.xml (deflated 41%)
2022-09-27T16:19:57.3523601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154437.xml (deflated 40%)
2022-09-27T16:19:57.3524188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154440.xml (deflated 42%)
2022-09-27T16:19:57.3524765Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154444.xml (deflated 41%)
2022-09-27T16:19:57.3525350Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154446.xml (deflated 41%)
2022-09-27T16:19:57.3525934Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154447.xml (deflated 42%)
2022-09-27T16:19:57.3526515Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154449.xml (deflated 41%)
2022-09-27T16:19:57.3527091Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154450.xml (deflated 42%)
2022-09-27T16:19:57.3527675Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154452.xml (deflated 42%)
2022-09-27T16:19:57.3528310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154456.xml (deflated 41%)
2022-09-27T16:19:57.3528898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154459.xml (deflated 42%)
2022-09-27T16:19:57.3529468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154501.xml (deflated 41%)
2022-09-27T16:19:57.3530057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154502.xml (deflated 42%)
2022-09-27T16:19:57.3530645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154506.xml (deflated 41%)
2022-09-27T16:19:57.3531233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154510.xml (deflated 41%)
2022-09-27T16:19:57.3531818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154512.xml (deflated 42%)
2022-09-27T16:19:57.3532447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154515.xml (deflated 41%)
2022-09-27T16:19:57.3533033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154519.xml (deflated 42%)
2022-09-27T16:19:57.3533617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154523.xml (deflated 41%)
2022-09-27T16:19:57.3534192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154527.xml (deflated 41%)
2022-09-27T16:19:57.3534774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154531.xml (deflated 42%)
2022-09-27T16:19:57.3535363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154532.xml (deflated 42%)
2022-09-27T16:19:57.3535951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154534.xml (deflated 41%)
2022-09-27T16:19:57.3536529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154536.xml (deflated 40%)
2022-09-27T16:19:57.3537112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154540.xml (deflated 41%)
2022-09-27T16:19:57.3537696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154543.xml (deflated 42%)
2022-09-27T16:19:57.3538279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154547.xml (deflated 41%)
2022-09-27T16:19:57.3538854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154549.xml (deflated 40%)
2022-09-27T16:19:57.3539448Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154552.xml (deflated 41%)
2022-09-27T16:19:57.3540033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154556.xml (deflated 41%)
2022-09-27T16:19:57.3540621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154600.xml (deflated 42%)
2022-09-27T16:19:57.3541195Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154602.xml (deflated 40%)
2022-09-27T16:19:57.3541781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154606.xml (deflated 40%)
2022-09-27T16:19:57.3542368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154609.xml (deflated 40%)
2022-09-27T16:19:57.3542997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154613.xml (deflated 42%)
2022-09-27T16:19:57.3543568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154615.xml (deflated 41%)
2022-09-27T16:19:57.3544361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154619.xml (deflated 40%)
2022-09-27T16:19:57.3544959Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154623.xml (deflated 41%)
2022-09-27T16:19:57.3545549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154627.xml (deflated 40%)
2022-09-27T16:19:57.3546125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154630.xml (deflated 40%)
2022-09-27T16:19:57.3546773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154634.xml (deflated 41%)
2022-09-27T16:19:57.3547371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154636.xml (deflated 40%)
2022-09-27T16:19:57.3547961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154640.xml (deflated 41%)
2022-09-27T16:19:57.3548551Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154644.xml (deflated 41%)
2022-09-27T16:19:57.3549132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154647.xml (deflated 41%)
2022-09-27T16:19:57.3549720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154651.xml (deflated 41%)
2022-09-27T16:19:57.3550315Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154653.xml (deflated 41%)
2022-09-27T16:19:57.3550913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154654.xml (deflated 41%)
2022-09-27T16:19:57.3551494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154700.xml (deflated 41%)
2022-09-27T16:19:57.3552083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml (deflated 40%)
2022-09-27T16:19:57.3552688Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154710.xml (deflated 40%)
2022-09-27T16:19:57.3553275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154714.xml (deflated 40%)
2022-09-27T16:19:57.3553853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154718.xml (deflated 42%)
2022-09-27T16:19:57.3554446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154719.xml (deflated 42%)
2022-09-27T16:19:57.3555036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154721.xml (deflated 42%)
2022-09-27T16:19:57.3555628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154722.xml (deflated 42%)
2022-09-27T16:19:57.3556206Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154724.xml (deflated 43%)
2022-09-27T16:19:57.3556794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154725.xml (deflated 42%)
2022-09-27T16:19:57.3557381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154727.xml (deflated 42%)
2022-09-27T16:19:57.3558024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154728.xml (deflated 42%)
2022-09-27T16:19:57.3558607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154730.xml (deflated 41%)
2022-09-27T16:19:57.3559196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154734.xml (deflated 42%)
2022-09-27T16:19:57.3559783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154737.xml (deflated 41%)
2022-09-27T16:19:57.3560375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154739.xml (deflated 43%)
2022-09-27T16:19:57.3560954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154741.xml (deflated 42%)
2022-09-27T16:19:57.3561584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154742.xml (deflated 42%)
2022-09-27T16:19:57.3562181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154746.xml (deflated 42%)
2022-09-27T16:19:57.3562774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154750.xml (deflated 41%)
2022-09-27T16:19:57.3563351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154754.xml (deflated 41%)
2022-09-27T16:19:57.3563938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154757.xml (deflated 41%)
2022-09-27T16:19:57.3564527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154801.xml (deflated 41%)
2022-09-27T16:19:57.3565119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154805.xml (deflated 40%)
2022-09-27T16:19:57.3565699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154809.xml (deflated 41%)
2022-09-27T16:19:57.3566284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154812.xml (deflated 40%)
2022-09-27T16:19:57.3566869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154816.xml (deflated 41%)
2022-09-27T16:19:57.3567457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154820.xml (deflated 40%)
2022-09-27T16:19:57.3568032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154824.xml (deflated 40%)
2022-09-27T16:19:57.3568621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154828.xml (deflated 40%)
2022-09-27T16:19:57.3569214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154832.xml (deflated 41%)
2022-09-27T16:19:57.3569804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154836.xml (deflated 40%)
2022-09-27T16:19:57.3570383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154840.xml (deflated 41%)
2022-09-27T16:19:57.3570970Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154844.xml (deflated 41%)
2022-09-27T16:19:57.3571555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154848.xml (deflated 41%)
2022-09-27T16:19:57.3572176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154851.xml (deflated 43%)
2022-09-27T16:19:57.3572750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154853.xml (deflated 41%)
2022-09-27T16:19:57.3573342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154857.xml (deflated 43%)
2022-09-27T16:19:57.3573930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154858.xml (deflated 41%)
2022-09-27T16:19:57.3574523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154902.xml (deflated 42%)
2022-09-27T16:19:57.3575096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154904.xml (deflated 42%)
2022-09-27T16:19:57.3575684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154905.xml (deflated 41%)
2022-09-27T16:19:57.3576299Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154909.xml (deflated 41%)
2022-09-27T16:19:57.3576892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154913.xml (deflated 41%)
2022-09-27T16:19:57.3577465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154917.xml (deflated 41%)
2022-09-27T16:19:57.3578048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154921.xml (deflated 41%)
2022-09-27T16:19:57.3578633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154922.xml (deflated 41%)
2022-09-27T16:19:57.3579220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154924.xml (deflated 41%)
2022-09-27T16:19:57.3579809Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154928.xml (deflated 41%)
2022-09-27T16:19:57.3580390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154932.xml (deflated 41%)
2022-09-27T16:19:57.3580978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154935.xml (deflated 41%)
2022-09-27T16:19:57.3581566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154939.xml (deflated 40%)
2022-09-27T16:19:57.3582155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154943.xml (deflated 40%)
2022-09-27T16:19:57.3582730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154947.xml (deflated 41%)
2022-09-27T16:19:57.3583320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154951.xml (deflated 41%)
2022-09-27T16:19:57.3584056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154955.xml (deflated 42%)
2022-09-27T16:19:57.3584736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154956.xml (deflated 41%)
2022-09-27T16:19:57.3585315Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154958.xml (deflated 41%)
2022-09-27T16:19:57.3585908Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154959.xml (deflated 41%)
2022-09-27T16:19:57.3586501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155003.xml (deflated 41%)
2022-09-27T16:19:57.3587172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155007.xml (deflated 41%)
2022-09-27T16:19:57.3587751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155011.xml (deflated 41%)
2022-09-27T16:19:57.3588339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155014.xml (deflated 41%)
2022-09-27T16:19:57.3588935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155018.xml (deflated 42%)
2022-09-27T16:19:57.3589526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155022.xml (deflated 42%)
2022-09-27T16:19:57.3590097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155026.xml (deflated 41%)
2022-09-27T16:19:57.3590727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155029.xml (deflated 41%)
2022-09-27T16:19:57.3591318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155033.xml (deflated 42%)
2022-09-27T16:19:57.3591906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155037.xml (deflated 41%)
2022-09-27T16:19:57.3592480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155038.xml (deflated 41%)
2022-09-27T16:19:57.3593062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155041.xml (deflated 42%)
2022-09-27T16:19:57.3593655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155045.xml (deflated 42%)
2022-09-27T16:19:57.3594243Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155046.xml (deflated 42%)
2022-09-27T16:19:57.3594823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155050.xml (deflated 43%)
2022-09-27T16:19:57.3595417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155054.xml (deflated 41%)
2022-09-27T16:19:57.3596004Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155058.xml (deflated 41%)
2022-09-27T16:19:57.3596591Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155102.xml (deflated 41%)
2022-09-27T16:19:57.3597216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155106.xml (deflated 41%)
2022-09-27T16:19:57.3597801Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155109.xml (deflated 41%)
2022-09-27T16:19:57.3598388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155113.xml (deflated 41%)
2022-09-27T16:19:57.3598986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml (deflated 40%)
2022-09-27T16:19:57.3599561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155118.xml (deflated 41%)
2022-09-27T16:19:57.3600149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155122.xml (deflated 41%)
2022-09-27T16:19:57.3600739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155126.xml (deflated 42%)
2022-09-27T16:19:57.3601329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155127.xml (deflated 41%)
2022-09-27T16:19:57.3601940Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155131.xml (deflated 42%)
2022-09-27T16:19:57.3602533Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155135.xml (deflated 42%)
2022-09-27T16:19:57.3603121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml (deflated 42%)
2022-09-27T16:19:57.3603708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155140.xml (deflated 46%)
2022-09-27T16:19:57.3604282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml (deflated 47%)
2022-09-27T16:19:57.3604874Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155148.xml (deflated 48%)
2022-09-27T16:19:57.3605502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155152.xml (deflated 46%)
2022-09-27T16:19:57.3606097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155156.xml (deflated 40%)
2022-09-27T16:19:57.3606684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155159.xml (deflated 41%)
2022-09-27T16:19:57.3607255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155203.xml (deflated 41%)
2022-09-27T16:19:57.3607842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml (deflated 41%)
2022-09-27T16:19:57.3608425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155211.xml (deflated 41%)
2022-09-27T16:19:57.3609019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155215.xml (deflated 41%)
2022-09-27T16:19:57.3609588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml (deflated 40%)
2022-09-27T16:19:57.3610176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155222.xml (deflated 43%)
2022-09-27T16:19:57.3610761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155224.xml (deflated 41%)
2022-09-27T16:19:57.3611345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155226.xml (deflated 41%)
2022-09-27T16:19:57.3611933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155229.xml (deflated 41%)
2022-09-27T16:19:57.3612523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155233.xml (deflated 43%)
2022-09-27T16:19:57.3613112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155235.xml (deflated 42%)
2022-09-27T16:19:57.3613704Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155236.xml (deflated 40%)
2022-09-27T16:19:57.3614279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155240.xml (deflated 40%)
2022-09-27T16:19:57.3614926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155244.xml (deflated 43%)
2022-09-27T16:19:57.3615859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155246.xml (deflated 41%)
2022-09-27T16:19:57.3616863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155250.xml (deflated 41%)
2022-09-27T16:19:57.3617591Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155253.xml (deflated 41%)
2022-09-27T16:19:57.3618188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155257.xml (deflated 41%)
2022-09-27T16:19:57.3618786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155301.xml (deflated 40%)
2022-09-27T16:19:57.3619374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155305.xml (deflated 40%)
2022-09-27T16:19:57.3619954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155309.xml (deflated 41%)
2022-09-27T16:19:57.3620545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155313.xml (deflated 41%)
2022-09-27T16:19:57.3621177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml (deflated 41%)
2022-09-27T16:19:57.3621871Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155321.xml (deflated 41%)
2022-09-27T16:19:57.3622460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml (deflated 41%)
2022-09-27T16:19:57.3623049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155328.xml (deflated 41%)
2022-09-27T16:19:57.3623630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155332.xml (deflated 41%)
2022-09-27T16:19:57.3624415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155336.xml (deflated 41%)
2022-09-27T16:19:57.3625020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155340.xml (deflated 40%)
2022-09-27T16:19:57.3625613Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155344.xml (deflated 40%)
2022-09-27T16:19:57.3626190Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155348.xml (deflated 41%)
2022-09-27T16:19:57.3626782Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155351.xml (deflated 40%)
2022-09-27T16:19:57.3627371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155355.xml (deflated 40%)
2022-09-27T16:19:57.3627961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155359.xml (deflated 40%)
2022-09-27T16:19:57.3628541Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155403.xml (deflated 40%)
2022-09-27T16:19:57.3629128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155407.xml (deflated 40%)
2022-09-27T16:19:57.3629714Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155411.xml (deflated 41%)
2022-09-27T16:19:57.3630305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155415.xml (deflated 41%)
2022-09-27T16:19:57.3630883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155419.xml (deflated 42%)
2022-09-27T16:19:57.3631477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml (deflated 41%)
2022-09-27T16:19:57.3632148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155426.xml (deflated 41%)
2022-09-27T16:19:57.3632739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155430.xml (deflated 42%)
2022-09-27T16:19:57.3633331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155434.xml (deflated 41%)
2022-09-27T16:19:57.3633912Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155438.xml (deflated 41%)
2022-09-27T16:19:57.3634503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155442.xml (deflated 41%)
2022-09-27T16:19:57.3635093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155445.xml (deflated 42%)
2022-09-27T16:19:57.3635729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155449.xml (deflated 42%)
2022-09-27T16:19:57.3636307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155453.xml (deflated 42%)
2022-09-27T16:19:57.3636896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155457.xml (deflated 43%)
2022-09-27T16:19:57.3637488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155458.xml (deflated 42%)
2022-09-27T16:19:57.3638071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155500.xml (deflated 43%)
2022-09-27T16:19:57.3655047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155501.xml (deflated 42%)
2022-09-27T16:19:57.3655669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155503.xml (deflated 43%)
2022-09-27T16:19:57.3656284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155504.xml (deflated 42%)
2022-09-27T16:19:57.3656877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155506.xml (deflated 43%)
2022-09-27T16:19:57.3657481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155507.xml (deflated 42%)
2022-09-27T16:19:57.3658076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml (deflated 43%)
2022-09-27T16:19:57.3658676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155511.xml (deflated 42%)
2022-09-27T16:19:57.3659261Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml (deflated 43%)
2022-09-27T16:19:57.3659857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml (deflated 43%)
2022-09-27T16:19:57.3660453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155515.xml (deflated 43%)
2022-09-27T16:19:57.3661045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155517.xml (deflated 43%)
2022-09-27T16:19:57.3661625Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155518.xml (deflated 42%)
2022-09-27T16:19:57.3662221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155520.xml (deflated 43%)
2022-09-27T16:19:57.3662813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155521.xml (deflated 43%)
2022-09-27T16:19:57.3663513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155523.xml (deflated 42%)
2022-09-27T16:19:57.3664236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155524.xml (deflated 43%)
2022-09-27T16:19:57.3664899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155526.xml (deflated 42%)
2022-09-27T16:19:57.3665497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155528.xml (deflated 42%)
2022-09-27T16:19:57.3666090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155529.xml (deflated 43%)
2022-09-27T16:19:57.3666673Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155531.xml (deflated 43%)
2022-09-27T16:19:57.3667340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155532.xml (deflated 42%)
2022-09-27T16:19:57.3667934Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155534.xml (deflated 42%)
2022-09-27T16:19:57.3668527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155538.xml (deflated 42%)
2022-09-27T16:19:57.3669104Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155539.xml (deflated 42%)
2022-09-27T16:19:57.3669692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155541.xml (deflated 41%)
2022-09-27T16:19:57.3670283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155546.xml (deflated 42%)
2022-09-27T16:19:57.3670876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml (deflated 40%)
2022-09-27T16:19:57.3671466Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155555.xml (deflated 42%)
2022-09-27T16:19:57.3672045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155558.xml (deflated 40%)
2022-09-27T16:19:57.3672635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155603.xml (deflated 41%)
2022-09-27T16:19:57.3673223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155607.xml (deflated 40%)
2022-09-27T16:19:57.3673814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155612.xml (deflated 41%)
2022-09-27T16:19:57.3674398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155617.xml (deflated 41%)
2022-09-27T16:19:57.3674991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155626.xml (deflated 40%)
2022-09-27T16:19:57.3675578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155629.xml (deflated 40%)
2022-09-27T16:19:57.3676168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml (deflated 41%)
2022-09-27T16:19:57.3676746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155635.xml (deflated 41%)
2022-09-27T16:19:57.3677331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155636.xml (deflated 41%)
2022-09-27T16:19:57.3677977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155638.xml (deflated 41%)
2022-09-27T16:19:57.3678568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155640.xml (deflated 41%)
2022-09-27T16:19:57.3679145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155641.xml (deflated 41%)
2022-09-27T16:19:57.3679738Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml (deflated 41%)
2022-09-27T16:19:57.3680325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155644.xml (deflated 41%)
2022-09-27T16:19:57.3680907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155646.xml (deflated 41%)
2022-09-27T16:19:57.3681489Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml (deflated 42%)
2022-09-27T16:19:57.3682108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155651.xml (deflated 40%)
2022-09-27T16:19:57.3682698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml (deflated 40%)
2022-09-27T16:19:57.3683285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155659.xml (deflated 42%)
2022-09-27T16:19:57.3683867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155703.xml (deflated 42%)
2022-09-27T16:19:57.3684452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml (deflated 41%)
2022-09-27T16:19:57.3685036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155706.xml (deflated 41%)
2022-09-27T16:19:57.3685630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155708.xml (deflated 42%)
2022-09-27T16:19:57.3686209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml (deflated 41%)
2022-09-27T16:19:57.3686791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155716.xml (deflated 41%)
2022-09-27T16:19:57.3687381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155719.xml (deflated 41%)
2022-09-27T16:19:57.3687967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155723.xml (deflated 41%)
2022-09-27T16:19:57.3688543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155725.xml (deflated 41%)
2022-09-27T16:19:57.3689134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155728.xml (deflated 41%)
2022-09-27T16:19:57.3689724Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml (deflated 41%)
2022-09-27T16:19:57.3690311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155734.xml (deflated 41%)
2022-09-27T16:19:57.3690886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml (deflated 41%)
2022-09-27T16:19:57.3691472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155739.xml (deflated 41%)
2022-09-27T16:19:57.3692058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155741.xml (deflated 41%)
2022-09-27T16:19:57.3692683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155742.xml (deflated 42%)
2022-09-27T16:19:57.3693259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155746.xml (deflated 41%)
2022-09-27T16:19:57.3693846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155750.xml (deflated 41%)
2022-09-27T16:19:57.3694430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155754.xml (deflated 41%)
2022-09-27T16:19:57.3695015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155758.xml (deflated 41%)
2022-09-27T16:19:57.3695591Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml (deflated 41%)
2022-09-27T16:19:57.3696214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155805.xml (deflated 42%)
2022-09-27T16:19:57.3696806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml (deflated 42%)
2022-09-27T16:19:57.3697389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155812.xml (deflated 42%)
2022-09-27T16:19:57.3697962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml (deflated 42%)
2022-09-27T16:19:57.3698553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155820.xml (deflated 42%)
2022-09-27T16:19:57.3699139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155823.xml (deflated 41%)
2022-09-27T16:19:57.3699719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155827.xml (deflated 42%)
2022-09-27T16:19:57.3700294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155831.xml (deflated 41%)
2022-09-27T16:19:57.3700874Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155834.xml (deflated 42%)
2022-09-27T16:19:57.3701459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155838.xml (deflated 42%)
2022-09-27T16:19:57.3702040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155842.xml (deflated 42%)
2022-09-27T16:19:57.3702614Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155846.xml (deflated 42%)
2022-09-27T16:19:57.3703192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155849.xml (deflated 41%)
2022-09-27T16:19:57.3703783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155851.xml (deflated 42%)
2022-09-27T16:19:57.3704590Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155855.xml (deflated 41%)
2022-09-27T16:19:57.3705172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155858.xml (deflated 40%)
2022-09-27T16:19:57.3705760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155902.xml (deflated 42%)
2022-09-27T16:19:57.3706351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155906.xml (deflated 41%)
2022-09-27T16:19:57.3706929Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155908.xml (deflated 41%)
2022-09-27T16:19:57.3707610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155909.xml (deflated 42%)
2022-09-27T16:19:57.3708188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155911.xml (deflated 41%)
2022-09-27T16:19:57.3708772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155912.xml (deflated 42%)
2022-09-27T16:19:57.3709353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155914.xml (deflated 42%)
2022-09-27T16:19:57.3709940Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml (deflated 41%)
2022-09-27T16:19:57.3710517Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155921.xml (deflated 42%)
2022-09-27T16:19:57.3711141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155923.xml (deflated 42%)
2022-09-27T16:19:57.3711729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155925.xml (deflated 42%)
2022-09-27T16:19:57.3712317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155928.xml (deflated 41%)
2022-09-27T16:19:57.3712897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155932.xml (deflated 41%)
2022-09-27T16:19:57.3713483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155934.xml (deflated 42%)
2022-09-27T16:19:57.3714071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155937.xml (deflated 41%)
2022-09-27T16:19:57.3714664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155941.xml (deflated 42%)
2022-09-27T16:19:57.3715239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155945.xml (deflated 41%)
2022-09-27T16:19:57.3715823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155948.xml (deflated 41%)
2022-09-27T16:19:57.3716411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155952.xml (deflated 42%)
2022-09-27T16:19:57.3716993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155954.xml (deflated 42%)
2022-09-27T16:19:57.3717568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155956.xml (deflated 41%)
2022-09-27T16:19:57.3718158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155957.xml (deflated 41%)
2022-09-27T16:19:57.3718742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160001.xml (deflated 41%)
2022-09-27T16:19:57.3719323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160005.xml (deflated 42%)
2022-09-27T16:19:57.3719899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160009.xml (deflated 41%)
2022-09-27T16:19:57.3720485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160010.xml (deflated 41%)
2022-09-27T16:19:57.3721069Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160014.xml (deflated 41%)
2022-09-27T16:19:57.3721714Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160018.xml (deflated 41%)
2022-09-27T16:19:57.3722291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml (deflated 43%)
2022-09-27T16:19:57.3722876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160023.xml (deflated 40%)
2022-09-27T16:19:57.3723458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160027.xml (deflated 41%)
2022-09-27T16:19:57.3724047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160031.xml (deflated 40%)
2022-09-27T16:19:57.3724623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160035.xml (deflated 42%)
2022-09-27T16:19:57.3725215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160037.xml (deflated 41%)
2022-09-27T16:19:57.3725824Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160040.xml (deflated 41%)
2022-09-27T16:19:57.3726407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160044.xml (deflated 41%)
2022-09-27T16:19:57.3726983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160048.xml (deflated 40%)
2022-09-27T16:19:57.3727567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160052.xml (deflated 40%)
2022-09-27T16:19:57.3728150Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160056.xml (deflated 41%)
2022-09-27T16:19:57.3728733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160057.xml (deflated 41%)
2022-09-27T16:19:57.3729310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160101.xml (deflated 41%)
2022-09-27T16:19:57.3729895Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160105.xml (deflated 41%)
2022-09-27T16:19:57.3730477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160109.xml (deflated 41%)
2022-09-27T16:19:57.3731062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160113.xml (deflated 41%)
2022-09-27T16:19:57.3731634Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160115.xml (deflated 41%)
2022-09-27T16:19:57.3732222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160116.xml (deflated 42%)
2022-09-27T16:19:57.3732816Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160122.xml (deflated 41%)
2022-09-27T16:19:57.3733404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160128.xml (deflated 40%)
2022-09-27T16:19:57.3733977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160132.xml (deflated 40%)
2022-09-27T16:19:57.3734565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160136.xml (deflated 40%)
2022-09-27T16:19:57.3735149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160140.xml (deflated 42%)
2022-09-27T16:19:57.3735735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160141.xml (deflated 42%)
2022-09-27T16:19:57.3736342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160143.xml (deflated 42%)
2022-09-27T16:19:57.3736928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160144.xml (deflated 42%)
2022-09-27T16:19:57.3737519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160146.xml (deflated 43%)
2022-09-27T16:19:57.3738170Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160147.xml (deflated 42%)
2022-09-27T16:19:57.3738757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160149.xml (deflated 42%)
2022-09-27T16:19:57.3739337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160150.xml (deflated 42%)
2022-09-27T16:19:57.3739956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160152.xml (deflated 41%)
2022-09-27T16:19:57.3740548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160156.xml (deflated 42%)
2022-09-27T16:19:57.3741125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160200.xml (deflated 42%)
2022-09-27T16:19:57.3741709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160201.xml (deflated 43%)
2022-09-27T16:19:57.3742294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160203.xml (deflated 42%)
2022-09-27T16:19:57.3742879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160204.xml (deflated 42%)
2022-09-27T16:19:57.3743470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160208.xml (deflated 42%)
2022-09-27T16:19:57.3744161Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160212.xml (deflated 41%)
2022-09-27T16:19:57.3744819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160216.xml (deflated 41%)
2022-09-27T16:19:57.3745403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160220.xml (deflated 41%)
2022-09-27T16:19:57.3745992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml (deflated 41%)
2022-09-27T16:19:57.3746570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160227.xml (deflated 41%)
2022-09-27T16:19:57.3747161Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160231.xml (deflated 41%)
2022-09-27T16:19:57.3747749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160235.xml (deflated 40%)
2022-09-27T16:19:57.3748335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160238.xml (deflated 41%)
2022-09-27T16:19:57.3749062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160242.xml (deflated 40%)
2022-09-27T16:19:57.3749710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160246.xml (deflated 41%)
2022-09-27T16:19:57.3750299Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160250.xml (deflated 41%)
2022-09-27T16:19:57.3750889Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160254.xml (deflated 41%)
2022-09-27T16:19:57.3751532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160258.xml (deflated 40%)
2022-09-27T16:19:57.3752120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160302.xml (deflated 41%)
2022-09-27T16:19:57.3752709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160306.xml (deflated 41%)
2022-09-27T16:19:57.3753298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160310.xml (deflated 41%)
2022-09-27T16:19:57.3753876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160314.xml (deflated 43%)
2022-09-27T16:19:57.3754464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160315.xml (deflated 41%)
2022-09-27T16:19:57.3755084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160319.xml (deflated 43%)
2022-09-27T16:19:57.3755671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160321.xml (deflated 41%)
2022-09-27T16:19:57.3756243Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160324.xml (deflated 42%)
2022-09-27T16:19:57.3756828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160326.xml (deflated 42%)
2022-09-27T16:19:57.3757417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160328.xml (deflated 41%)
2022-09-27T16:19:57.3758005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160331.xml (deflated 41%)
2022-09-27T16:19:57.3758583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160335.xml (deflated 41%)
2022-09-27T16:19:57.3759170Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160339.xml (deflated 41%)
2022-09-27T16:19:57.3759757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160343.xml (deflated 41%)
2022-09-27T16:19:57.3760341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160344.xml (deflated 41%)
2022-09-27T16:19:57.3760915Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160346.xml (deflated 40%)
2022-09-27T16:19:57.3761499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160350.xml (deflated 41%)
2022-09-27T16:19:57.3762088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160354.xml (deflated 40%)
2022-09-27T16:19:57.3762676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160358.xml (deflated 41%)
2022-09-27T16:19:57.3763252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160401.xml (deflated 41%)
2022-09-27T16:19:57.3763837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160405.xml (deflated 41%)
2022-09-27T16:19:57.3764421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160409.xml (deflated 41%)
2022-09-27T16:19:57.3765009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160413.xml (deflated 40%)
2022-09-27T16:19:57.3765618Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml (deflated 42%)
2022-09-27T16:19:57.3766206Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160418.xml (deflated 41%)
2022-09-27T16:19:57.3766793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160420.xml (deflated 41%)
2022-09-27T16:19:57.3767383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160421.xml (deflated 41%)
2022-09-27T16:19:57.3767958Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160425.xml (deflated 41%)
2022-09-27T16:19:57.3768545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160429.xml (deflated 41%)
2022-09-27T16:19:57.3769131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160433.xml (deflated 40%)
2022-09-27T16:19:57.3769752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160436.xml (deflated 41%)
2022-09-27T16:19:57.3770330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160440.xml (deflated 42%)
2022-09-27T16:19:57.3770917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160444.xml (deflated 42%)
2022-09-27T16:19:57.3771502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160448.xml (deflated 40%)
2022-09-27T16:19:57.3772090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160452.xml (deflated 41%)
2022-09-27T16:19:57.3772663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160455.xml (deflated 42%)
2022-09-27T16:19:57.3773260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160459.xml (deflated 41%)
2022-09-27T16:19:57.3773851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160500.xml (deflated 41%)
2022-09-27T16:19:57.3774488Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161750.xml (deflated 42%)
2022-09-27T16:19:57.3775152Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161754.xml (deflated 41%)
2022-09-27T16:19:57.3775820Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161757.xml (deflated 42%)
2022-09-27T16:19:57.3776490Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161758.xml (deflated 42%)
2022-09-27T16:19:57.3777167Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161800.xml (deflated 42%)
2022-09-27T16:19:57.3777834Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161801.xml (deflated 42%)
2022-09-27T16:19:57.3778491Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161804.xml (deflated 41%)
2022-09-27T16:19:57.3779159Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161807.xml (deflated 41%)
2022-09-27T16:19:57.3779823Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161810.xml (deflated 42%)
2022-09-27T16:19:57.3780524Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161812.xml (deflated 42%)
2022-09-27T16:19:57.3781178Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161813.xml (deflated 42%)
2022-09-27T16:19:57.3781843Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927161815.xml (deflated 42%)
2022-09-27T16:19:57.3799102Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:19:57.3799407Z # Remove any previous test reports if they exist
2022-09-27T16:19:57.3799643Z rm -f usage-log-*.zip
2022-09-27T16:19:57.3799922Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-09-27T16:19:57.3800200Z # so check to see if the file exists first
2022-09-27T16:19:57.3800435Z if [ -f 'usage_log.txt' ]; then
2022-09-27T16:19:57.3800681Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-09-27T16:19:57.3800929Z fi
2022-09-27T16:19:57.3811294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:19:57.3811513Z env:
2022-09-27T16:19:57.3811679Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.3811933Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_8576161444
2022-09-27T16:19:57.3812170Z ##[endgroup]
2022-09-27T16:19:57.4191858Z   adding: usage_log.txt
2022-09-27T16:19:57.4192254Z 	zip warning:  file size changed while zipping usage_log.txt
2022-09-27T16:19:57.4192613Z  (deflated 95%)
2022-09-27T16:19:57.4227240Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:19:57.4227453Z with:
2022-09-27T16:19:57.4227655Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:19:57.4227874Z   retention-days: 14
2022-09-27T16:19:57.4228055Z   if-no-files-found: warn
2022-09-27T16:19:57.4228258Z   path: test-jsons-*.zip
2022-09-27T16:19:57.4228440Z   name: artifact
2022-09-27T16:19:57.4228608Z   s3-bucket: gha-artifacts
2022-09-27T16:19:57.4228805Z   region: us-east-1
2022-09-27T16:19:57.4228972Z env:
2022-09-27T16:19:57.4229127Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.4229313Z ##[endgroup]
2022-09-27T16:19:57.7246728Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:19:57.7247270Z With the provided path, there will be 1 file uploaded
2022-09-27T16:19:57.7247596Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:19:57.7253334Z Starting upload of test-jsons-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:57.8645624Z Finished upload of test-jsons-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:57.8765019Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:19:57.8765391Z with:
2022-09-27T16:19:57.8765729Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:19:57.8766075Z   retention-days: 14
2022-09-27T16:19:57.8766437Z   if-no-files-found: error
2022-09-27T16:19:57.8766796Z   path: test-reports-*.zip
2022-09-27T16:19:57.8767110Z   name: artifact
2022-09-27T16:19:57.8767431Z   s3-bucket: gha-artifacts
2022-09-27T16:19:57.8767758Z   region: us-east-1
2022-09-27T16:19:57.8768055Z env:
2022-09-27T16:19:57.8768347Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:57.8768676Z ##[endgroup]
2022-09-27T16:19:58.1855764Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:19:58.1856297Z With the provided path, there will be 1 file uploaded
2022-09-27T16:19:58.1856691Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:19:58.1863208Z Starting upload of test-reports-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:58.3948128Z Finished upload of test-reports-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:58.4057733Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:19:58.4057962Z with:
2022-09-27T16:19:58.4058251Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:19:58.4058458Z   retention-days: 14
2022-09-27T16:19:58.4058654Z   if-no-files-found: ignore
2022-09-27T16:19:58.4058859Z   path: usage-log-*.zip
2022-09-27T16:19:58.4059026Z   name: artifact
2022-09-27T16:19:58.4059464Z   s3-bucket: gha-artifacts
2022-09-27T16:19:58.4059656Z   region: us-east-1
2022-09-27T16:19:58.4059822Z env:
2022-09-27T16:19:58.4059981Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:58.4060168Z ##[endgroup]
2022-09-27T16:19:58.7081583Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:19:58.7081899Z With the provided path, there will be 1 file uploaded
2022-09-27T16:19:58.7082173Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:19:58.7089312Z Starting upload of usage-log-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:58.8446707Z Finished upload of usage-log-test-distributed-2-2-linux.2xlarge_8576161444.zip
2022-09-27T16:19:58.8557381Z ##[group]Run set -x
2022-09-27T16:19:58.8557572Z set -x
2022-09-27T16:19:58.8557791Z python3 -m pip install -r requirements.txt
2022-09-27T16:19:58.8558042Z python3 -m pip install boto3==1.19.12
2022-09-27T16:19:58.8558335Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:19:58.8569293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:19:58.8569511Z env:
2022-09-27T16:19:58.8569686Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:19:58.8569877Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T16:19:58.8570072Z   BRANCH: pull/85462
2022-09-27T16:19:58.8570259Z   TEST_CONFIG: distributed
2022-09-27T16:19:58.8570432Z   SHARD_NUMBER: 2
2022-09-27T16:19:58.8570648Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-09-27T16:19:58.8570863Z   PR_NUMBER: 85462
2022-09-27T16:19:58.8571038Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T16:19:58.8571245Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T16:19:58.8571502Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:19:58.8571689Z   TAG: 
2022-09-27T16:19:58.8571854Z   WORKFLOW_ID: 3133193930
2022-09-27T16:19:58.8572167Z   GITHUB_TOKEN: ***
2022-09-27T16:19:58.8572344Z   GHA_WORKFLOW_JOB_ID: 8576161444
2022-09-27T16:19:58.8572532Z ##[endgroup]
2022-09-27T16:19:58.8596680Z + python3 -m pip install -r requirements.txt
2022-09-27T16:19:59.0631044Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:19:59.0878602Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-09-27T16:19:59.0903207Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.3)
2022-09-27T16:19:59.0911171Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-09-27T16:19:59.0920306Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.54.6)
2022-09-27T16:19:59.1281733Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-09-27T16:19:59.1289357Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-09-27T16:19:59.1365486Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-09-27T16:19:59.1372690Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-09-27T16:19:59.1571942Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-09-27T16:19:59.1738249Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-09-27T16:19:59.1746537Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-09-27T16:19:59.1751965Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.3.0)
2022-09-27T16:19:59.1760800Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-09-27T16:19:59.1781836Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.37.1)
2022-09-27T16:19:59.1809349Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-09-27T16:19:59.1819087Z Requirement already satisfied: exceptiongroup>=1.0.0rc8; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.0rc9)
2022-09-27T16:19:59.1834718Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-09-27T16:19:59.2078104Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-09-27T16:19:59.2229754Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-09-27T16:19:59.2238107Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-09-27T16:19:59.2255805Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-09-27T16:19:59.2266674Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-09-27T16:19:59.2827486Z + python3 -m pip install boto3==1.19.12
2022-09-27T16:19:59.4878032Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:19:59.5054365Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-09-27T16:19:59.5102649Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-09-27T16:19:59.5144265Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-09-27T16:19:59.5176865Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-09-27T16:19:59.5189310Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-09-27T16:19:59.5208761Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-09-27T16:19:59.5359717Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-09-27T16:19:59.6897910Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:20:07.3985398Z [scribe] Scribe access token not provided, sending report via boto3...
2022-09-27T16:20:07.3985715Z 
2022-09-27T16:20:07.3986084Z ----- Historic stats comparison result ------
2022-09-27T16:20:07.3986356Z 
2022-09-27T16:20:07.3986861Z     job: linux-focal-py3.7-gcc7
2022-09-27T16:20:07.3987188Z     commit: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:20:07.3987373Z 
2022-09-27T16:20:07.3987630Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-09-27T16:20:07.3987958Z 
2022-09-27T16:20:07.3988082Z     : (master)
2022-09-27T16:20:07.3988294Z     |
2022-09-27T16:20:07.3988476Z     | * 52424e2bf3 (HEAD)            total time  1381.80s
2022-09-27T16:20:07.3988668Z     | |
2022-09-27T16:20:07.3988822Z     | : (4 commits)
2022-09-27T16:20:07.3988968Z     |/
2022-09-27T16:20:07.3990893Z     * c7c2578f93 (base)   6 reports, total time  2509.28s ± 1765.63s
2022-09-27T16:20:07.3991275Z     * 99ad8a3048          6 reports, total time  2509.77s ± 1771.89s
2022-09-27T16:20:07.3991579Z     * 34296e2f4c          6 reports, total time  2525.78s ± 1741.22s
2022-09-27T16:20:07.3992046Z     * 4523ac7aa1          6 reports, total time  2546.75s ± 1751.11s
2022-09-27T16:20:07.3992377Z     * f21e77d9a6          6 reports, total time  2521.77s ± 1744.10s
2022-09-27T16:20:07.3992701Z     * 26a861cb27          6 reports, total time  2538.71s ± 1776.59s
2022-09-27T16:20:07.3992999Z     * 56a41b5998          6 reports, total time  2537.11s ± 1781.85s
2022-09-27T16:20:07.3993311Z     * 1910c5847e          6 reports, total time  2539.59s ± 1745.17s
2022-09-27T16:20:07.3993627Z     * caa0ab557d          6 reports, total time  2502.08s ± 1718.61s
2022-09-27T16:20:07.3993832Z     * 0336308be5          0 reports
2022-09-27T16:20:07.3994002Z     |
2022-09-27T16:20:07.3994152Z     :
2022-09-27T16:20:07.3994242Z 
2022-09-27T16:20:07.3994348Z Removed  (across  823 suites)     0 tests, totaling     0.00s
2022-09-27T16:20:07.3994606Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-09-27T16:20:07.3994864Z Added    (across   73 suites)   970 tests, totaling +1795.85s
2022-09-27T16:20:07.4492354Z Prepare all required actions
2022-09-27T16:20:07.4513080Z ##[group]Run ./.github/actions/teardown-linux
2022-09-27T16:20:07.4513299Z with:
2022-09-27T16:20:07.4513442Z env:
2022-09-27T16:20:07.4513621Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:20:07.4513808Z ##[endgroup]
2022-09-27T16:20:07.4531066Z ##[group]Run set -eou pipefail
2022-09-27T16:20:07.4531289Z set -eou pipefail
2022-09-27T16:20:07.4531471Z 
2022-09-27T16:20:07.4531694Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-09-27T16:20:07.4531969Z for _ in $(seq 1440); do
2022-09-27T16:20:07.4532290Z     # Break if no ssh session exists anymore
2022-09-27T16:20:07.4532501Z     if [ "$(who)" = "" ]; then
2022-09-27T16:20:07.4532684Z       break
2022-09-27T16:20:07.4532850Z     fi
2022-09-27T16:20:07.4533017Z     echo "."
2022-09-27T16:20:07.4533177Z     sleep 5
2022-09-27T16:20:07.4533374Z done
2022-09-27T16:20:07.4544785Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:20:07.4545011Z env:
2022-09-27T16:20:07.4545189Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:20:07.4545364Z ##[endgroup]
2022-09-27T16:20:07.4569408Z Holding runner for 2 hours until all ssh sessions have logged out
2022-09-27T16:20:07.4606759Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:20:07.4607077Z # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:20:07.4607333Z # shellcheck disable=SC2046
2022-09-27T16:20:07.4607550Z docker stop $(docker ps -q) || true
2022-09-27T16:20:07.4607784Z # Prune all of the docker images
2022-09-27T16:20:07.4608004Z docker system prune -af
2022-09-27T16:20:07.4617885Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:20:07.4618110Z env:
2022-09-27T16:20:07.4618293Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:20:07.4618476Z ##[endgroup]
2022-09-27T16:20:07.8412227Z 89d4c2f4be45
2022-09-27T16:20:08.3707140Z Deleted Containers:
2022-09-27T16:20:08.3708066Z 89d4c2f4be4530632f6fcecdaeba7f9c867a1bfa7fb3dd5b99f5b3cc80af06b2
2022-09-27T16:20:08.3708454Z 
2022-09-27T16:20:10.8842841Z Deleted Images:
2022-09-27T16:20:10.8843713Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T16:20:10.8844516Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:aa31a48de2d62af84cacd725abaa1599e63c6bb3385fa6e176b6d91f90a39348
2022-09-27T16:20:10.8845446Z deleted: sha256:09cc5e0dbbe1120475828f67580c1e4ef0ac166558f7a9e82119c9daf09ed17b
2022-09-27T16:20:10.8845960Z deleted: sha256:4f4d0a1da9af50d9a8bd02155aaea8b254a8039b940db877e78a5b25776e1f97
2022-09-27T16:20:10.8846403Z deleted: sha256:40273608be9f35547c534c65183771d8d024a327b2f8231776e449a0c4be59e4
2022-09-27T16:20:10.8846736Z deleted: sha256:295b500c49641b27efb539c901bd567f2b8738ab1f67465c28c5209fef872936
2022-09-27T16:20:10.8847064Z deleted: sha256:aff7d79df966d7c1c7cf29b78408592957f0317054355d087aa59098aa32abda
2022-09-27T16:20:10.8847386Z deleted: sha256:0e0419c0016bdfbf589e8507dad2150575b0a84dd2b27be7703cfa05e2139c5b
2022-09-27T16:20:10.8847721Z deleted: sha256:0e1f7b9f93cf30ea54c8e6f1485a1a3fd9776d17cc90f1bfb5ce6b5b9185296e
2022-09-27T16:20:10.8848072Z deleted: sha256:d3deacfa23bfb0920ec3a625c1ee1963513159aaa5463910dd41bafa8e54de71
2022-09-27T16:20:10.8848504Z deleted: sha256:e66d036a902f9f1d15a19abacd7321bba3d8869b725e99b43a5d20fd6fdb01c9
2022-09-27T16:20:10.8848900Z deleted: sha256:c5e59df58f1aab354a2874070ad89a043c3178016eab17887ed9237f91887294
2022-09-27T16:20:10.8849229Z deleted: sha256:0d8f363a24f0b380db49f8fa69e9da0537d3a51ae051248dfd3563de574ca15d
2022-09-27T16:20:10.8849568Z deleted: sha256:55bb4ae9fce9854a9f8345d077cd37423d7b6f050034cadadbbf1974ff155061
2022-09-27T16:20:10.8850024Z deleted: sha256:124538c5717f1c1615e490bfd34cab23dc00516fbd43b4dbd0755e1a5119474b
2022-09-27T16:20:10.8850350Z deleted: sha256:98e71d41b4fb04ffa587728b6f070c0d922a9909fb272edae447c0ed732b17c6
2022-09-27T16:20:10.8850678Z deleted: sha256:dd100ce53375a22379598335be3a232a3eb20465df4376afdd481c3ab0a01666
2022-09-27T16:20:10.8851058Z deleted: sha256:06da69152df74c7a52462b8f1f7adf2d50e5f84ae02c20a48597f844e24832f4
2022-09-27T16:20:10.8851382Z deleted: sha256:7f90b14cc4e879de499e9c152c229852c1a71e1c6bb0216e21b433d34e401be0
2022-09-27T16:20:10.8851704Z deleted: sha256:2a332f12721f3199efd9441306d5d38a6f031cdfdf68beb90afb40116f7ac1ad
2022-09-27T16:20:10.8852030Z deleted: sha256:9c866de535edd7886f74437d3fbcafd6d58fe3322aa4af8606e46e720f022e9c
2022-09-27T16:20:10.8852346Z deleted: sha256:097a48bd86a6a3542f566d38833d93ded3bc082bdad7e4026debfbe895bb5105
2022-09-27T16:20:10.8852787Z deleted: sha256:a8b120c70115f75a61a81c96d1abef9e741618d32124b4994375a805a5941958
2022-09-27T16:20:10.8853106Z deleted: sha256:15a928b5307c0332ba50caffe867123de36c74475d7861b8cdf2dbfd58dd5320
2022-09-27T16:20:10.8853439Z deleted: sha256:eb93d3b8365b2da69ddeb34c24e5ba2fae6eb2fad4b6fadae3e298a9a0b26af2
2022-09-27T16:20:10.8853791Z deleted: sha256:e2ab13aa2f895ecf403ea9781db734f7b21a20aeb1478e32c5416f969604dc89
2022-09-27T16:20:10.8854111Z deleted: sha256:6faa89577c8237b30a828e6382086b71579a76333d2d79839246ed4c42daf433
2022-09-27T16:20:10.8854425Z deleted: sha256:94f3fc0358a54e1bab1cff99f032187d852d5a9e21539debb385d3e965e0f811
2022-09-27T16:20:10.8854805Z deleted: sha256:0890fc3c642cc4c11d7b3b400a3f6af80996973f0d1524071eb15cbdd5431266
2022-09-27T16:20:10.8855131Z deleted: sha256:ec7d767c32abaf566f084ffeb2d37bda985f13102952e21948f7b055ccccf7ab
2022-09-27T16:20:10.8855453Z deleted: sha256:2758607d82553d625e21563dec5e252097fd6804de79dd60b78d9452d8f0e384
2022-09-27T16:20:10.8855748Z deleted: sha256:1c30230d1bc09ce084c2c0d3c24c81e39a270b3359be353f188f13809b2e95ff
2022-09-27T16:20:10.8856267Z deleted: sha256:f66b31a4878493b2a4cf4b457a461d78c1275ce0b853312e14b5b205f5f54b12
2022-09-27T16:20:10.8856674Z deleted: sha256:c96b1aa08b8cdf00ede17699be953c554e540b97216fba81a59a101860c877c6
2022-09-27T16:20:10.8857000Z deleted: sha256:0853e57dc641a08006c8b8cd96caa8755d3981706b49748776bf6c7ee23abe6c
2022-09-27T16:20:10.8857713Z deleted: sha256:a88ebc418ab3040ebaafd029a4c80439225b1ce939f383ec70c503fff55528ee
2022-09-27T16:20:10.8858290Z deleted: sha256:5a86190b95485dba13fec16a89446426425223587ad649a56b4faf0f64305120
2022-09-27T16:20:10.8858817Z deleted: sha256:016eff1a025a65973028af96cf1fbe6c4b0e1c258e04e6a2dec976062924c0cf
2022-09-27T16:20:10.8859295Z deleted: sha256:3ce5ce7ae3df63641c9b8859e93e8f870c8bcfadc998c17b2169676be5e0812d
2022-09-27T16:20:10.8859624Z deleted: sha256:b161ee8936514898f92131df4d7de87314f3ed0639e31bbd2c3e83bdea170f16
2022-09-27T16:20:10.8859948Z deleted: sha256:e6a3a6ca01553128af3686d5bface6135e56f897e0f9ba754e9b924d446105e6
2022-09-27T16:20:10.8860280Z deleted: sha256:0cd4c1b082ed42bb2648c766b0746e3eb8771eafc8ce4130c649f3d29b634162
2022-09-27T16:20:10.8860579Z deleted: sha256:76a601a88697897790d443b0214d908f489395a5d4ea98726caf0a80a66ecd19
2022-09-27T16:20:10.8860886Z deleted: sha256:e9a0ce4d7b4ea637152093c92c0cbe039a7e810c4aa90c848f312c10262b974d
2022-09-27T16:20:10.8861225Z deleted: sha256:e4e7cd35b02c90bb1da4c482c1f10f82c7ef6daae84f9ceeac9815d19ce51dbd
2022-09-27T16:20:10.8861556Z deleted: sha256:8e15ae424f8839bb14c5629f1fec3c93cebec42642eabeabd208ca359a19db66
2022-09-27T16:20:10.8861891Z deleted: sha256:e43a9a37dcae4c22f906f9f4450ad78b70905cd289f7ad9f9d40d7c8b12d8f72
2022-09-27T16:20:10.8862211Z deleted: sha256:06bcf3094318f05722b94bc725806fc0d2367a7865af5b199f0567a72699f757
2022-09-27T16:20:10.8862528Z deleted: sha256:51f25a037328b3be1af44473dfa7a876a5355f74cef818b46bf003c33b1ec2c5
2022-09-27T16:20:10.8862840Z deleted: sha256:b17dd3db6bb4f4513025c933f6a63f1ed0921065fe34fd195c40c4acff4f696c
2022-09-27T16:20:10.8863151Z deleted: sha256:71ca3ec58848585b585c80e00986d5e8789e8169554706949b8a3d8dcdf72e3b
2022-09-27T16:20:10.8863464Z deleted: sha256:6f4b9ef5ee757f0395c054927ff65df65438bdf23c05109f7f3107fb94719eef
2022-09-27T16:20:10.8864041Z deleted: sha256:883eb0bcef572233ec6334ad982f16d8ba42bcffee0735ae8969fe257019481d
2022-09-27T16:20:10.8864472Z deleted: sha256:f0d75587c52d51e623884d33f223de82faa346a03a9e34271eaf6765e8995d2a
2022-09-27T16:20:10.8864794Z deleted: sha256:25ab06e14d5870622fe0c785fa28eedb95787ed7286d8c665289672a2e11334a
2022-09-27T16:20:10.8865379Z deleted: sha256:f9bb5b08ac5635a4a64adbfe88fe48bcb0ed9425ab68ecf412265012933639e6
2022-09-27T16:20:10.8865882Z deleted: sha256:aa9ab2200af3ee6cbe32d9703272a4a85de2def9b0c9a20bd09f3dca10f0e47d
2022-09-27T16:20:10.8866255Z deleted: sha256:30936d838a38cd0a0c2ddc832a2a17af9d84327e0f23cac0e773686bfed7e6e1
2022-09-27T16:20:10.8866594Z deleted: sha256:5c5425a2d9bb189d0f99b1c37511d370d599afca6b8b0d3348b3b2ace341ddaf
2022-09-27T16:20:10.8867064Z deleted: sha256:998fb9c2fa230ea5cc1f997926614d854d193df0b2129b2b12d7ede4e5d43f6e
2022-09-27T16:20:10.8867666Z deleted: sha256:b40ed86654e59e1012e1716d5384910f8c3bb58274b7b00fca564a53e9897ba3
2022-09-27T16:20:10.8867928Z 
2022-09-27T16:20:10.8868047Z Total reclaimed space: 7.17GB
2022-09-27T16:20:10.8920180Z Post job cleanup.
2022-09-27T16:20:10.8948970Z Post job cleanup.
2022-09-27T16:20:10.9900798Z [command]/usr/bin/git version
2022-09-27T16:20:10.9941435Z git version 2.37.1
2022-09-27T16:20:10.9982355Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5d5aa2ba-6af7-44ff-b593-0a2bd09b0e30' before making global git config changes
2022-09-27T16:20:10.9983072Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T16:20:10.9989453Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T16:20:11.0028981Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T16:20:11.0060999Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T16:20:11.0319456Z Entering 'android/libs/fbjni'
2022-09-27T16:20:11.0352800Z Entering 'third_party/FP16'
2022-09-27T16:20:11.0389028Z Entering 'third_party/FXdiv'
2022-09-27T16:20:11.0420906Z Entering 'third_party/NNPACK'
2022-09-27T16:20:11.0456209Z Entering 'third_party/QNNPACK'
2022-09-27T16:20:11.0489695Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:20:11.0524157Z Entering 'third_party/XNNPACK'
2022-09-27T16:20:11.0568134Z Entering 'third_party/benchmark'
2022-09-27T16:20:11.0603658Z Entering 'third_party/cpuinfo'
2022-09-27T16:20:11.0637442Z Entering 'third_party/cub'
2022-09-27T16:20:11.0671144Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:20:11.0710333Z Entering 'third_party/cutlass'
2022-09-27T16:20:11.0751026Z Entering 'third_party/eigen'
2022-09-27T16:20:11.0786418Z Entering 'third_party/fbgemm'
2022-09-27T16:20:11.0819895Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:20:11.0853895Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:20:11.0888260Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:20:11.0921964Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:20:11.0957303Z Entering 'third_party/flatbuffers'
2022-09-27T16:20:11.0993412Z Entering 'third_party/fmt'
2022-09-27T16:20:11.1027860Z Entering 'third_party/foxi'
2022-09-27T16:20:11.1062327Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:20:11.1097413Z Entering 'third_party/gloo'
2022-09-27T16:20:11.1131361Z Entering 'third_party/googletest'
2022-09-27T16:20:11.1165532Z Entering 'third_party/ideep'
2022-09-27T16:20:11.1199373Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:20:11.1233717Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:20:11.1273876Z Entering 'third_party/ios-cmake'
2022-09-27T16:20:11.1308119Z Entering 'third_party/ittapi'
2022-09-27T16:20:11.1341556Z Entering 'third_party/kineto'
2022-09-27T16:20:11.1375259Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:20:11.1409504Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:20:11.1444806Z Entering 'third_party/nccl/nccl'
2022-09-27T16:20:11.1479592Z Entering 'third_party/neon2sse'
2022-09-27T16:20:11.1513365Z Entering 'third_party/nlohmann'
2022-09-27T16:20:11.1548312Z Entering 'third_party/onnx'
2022-09-27T16:20:11.1594899Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:20:11.1627712Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:20:11.1662411Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:20:11.1695172Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:20:11.1733566Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:20:11.1766908Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:20:11.1799129Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:20:11.1837777Z Entering 'third_party/pocketfft'
2022-09-27T16:20:11.1871376Z Entering 'third_party/protobuf'
2022-09-27T16:20:11.1909693Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:20:11.1942494Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:20:11.1977020Z Entering 'third_party/psimd'
2022-09-27T16:20:11.2011630Z Entering 'third_party/pthreadpool'
2022-09-27T16:20:11.2045538Z Entering 'third_party/pybind11'
2022-09-27T16:20:11.2078803Z Entering 'third_party/python-enum'
2022-09-27T16:20:11.2112853Z Entering 'third_party/python-peachpy'
2022-09-27T16:20:11.2146005Z Entering 'third_party/python-six'
2022-09-27T16:20:11.2179844Z Entering 'third_party/sleef'
2022-09-27T16:20:11.2213998Z Entering 'third_party/tbb'
2022-09-27T16:20:11.2249090Z Entering 'third_party/tensorpipe'
2022-09-27T16:20:11.2282659Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:20:11.2316617Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:20:11.2348720Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:20:11.2381497Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:20:11.2414291Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:20:11.2449563Z Entering 'third_party/zstd'
2022-09-27T16:20:11.2500439Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T16:20:11.2525980Z http.https://github.com/.extraheader
2022-09-27T16:20:11.2533666Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-09-27T16:20:11.2566816Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T16:20:11.2819835Z Entering 'android/libs/fbjni'
2022-09-27T16:20:11.2839090Z http.https://github.com/.extraheader
2022-09-27T16:20:11.2865247Z Entering 'third_party/FP16'
2022-09-27T16:20:11.2884960Z http.https://github.com/.extraheader
2022-09-27T16:20:11.2911235Z Entering 'third_party/FXdiv'
2022-09-27T16:20:11.2930438Z http.https://github.com/.extraheader
2022-09-27T16:20:11.2957582Z Entering 'third_party/NNPACK'
2022-09-27T16:20:11.2977000Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3002976Z Entering 'third_party/QNNPACK'
2022-09-27T16:20:11.3022485Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3048592Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:20:11.3069411Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3095366Z Entering 'third_party/XNNPACK'
2022-09-27T16:20:11.3115277Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3151395Z Entering 'third_party/benchmark'
2022-09-27T16:20:11.3170409Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3197053Z Entering 'third_party/cpuinfo'
2022-09-27T16:20:11.3215821Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3241871Z Entering 'third_party/cub'
2022-09-27T16:20:11.3262183Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3287952Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:20:11.3307442Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3338471Z Entering 'third_party/cutlass'
2022-09-27T16:20:11.3358584Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3391319Z Entering 'third_party/eigen'
2022-09-27T16:20:11.3410757Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3439253Z Entering 'third_party/fbgemm'
2022-09-27T16:20:11.3458234Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3484249Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:20:11.3503581Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3529595Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:20:11.3549064Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3574776Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:20:11.3594574Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3620941Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:20:11.3640153Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3667574Z Entering 'third_party/flatbuffers'
2022-09-27T16:20:11.3686596Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3714125Z Entering 'third_party/fmt'
2022-09-27T16:20:11.3733324Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3759482Z Entering 'third_party/foxi'
2022-09-27T16:20:11.3779170Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3804831Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:20:11.3826329Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3853392Z Entering 'third_party/gloo'
2022-09-27T16:20:11.3873445Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3899904Z Entering 'third_party/googletest'
2022-09-27T16:20:11.3918572Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3945038Z Entering 'third_party/ideep'
2022-09-27T16:20:11.3964588Z http.https://github.com/.extraheader
2022-09-27T16:20:11.3990808Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:20:11.4010041Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4037679Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:20:11.4058243Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4090660Z Entering 'third_party/ios-cmake'
2022-09-27T16:20:11.4112265Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4138753Z Entering 'third_party/ittapi'
2022-09-27T16:20:11.4159456Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4186463Z Entering 'third_party/kineto'
2022-09-27T16:20:11.4206376Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4253845Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:20:11.4273083Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4298679Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:20:11.4318229Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4346359Z Entering 'third_party/nccl/nccl'
2022-09-27T16:20:11.4365051Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4393981Z Entering 'third_party/neon2sse'
2022-09-27T16:20:11.4413072Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4439073Z Entering 'third_party/nlohmann'
2022-09-27T16:20:11.4458872Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4486696Z Entering 'third_party/onnx'
2022-09-27T16:20:11.4506948Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4543761Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:20:11.4562995Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4589731Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:20:11.4609145Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4637143Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:20:11.4656280Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4682476Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:20:11.4700801Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4731465Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:20:11.4752773Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4779411Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:20:11.4798805Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4824584Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:20:11.4843864Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4874703Z Entering 'third_party/pocketfft'
2022-09-27T16:20:11.4893819Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4920074Z Entering 'third_party/protobuf'
2022-09-27T16:20:11.4939831Z http.https://github.com/.extraheader
2022-09-27T16:20:11.4969544Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:20:11.4988381Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5013768Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:20:11.5033003Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5059810Z Entering 'third_party/psimd'
2022-09-27T16:20:11.5078881Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5106039Z Entering 'third_party/pthreadpool'
2022-09-27T16:20:11.5124331Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5150578Z Entering 'third_party/pybind11'
2022-09-27T16:20:11.5169977Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5195869Z Entering 'third_party/python-enum'
2022-09-27T16:20:11.5215141Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5240988Z Entering 'third_party/python-peachpy'
2022-09-27T16:20:11.5260306Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5287070Z Entering 'third_party/python-six'
2022-09-27T16:20:11.5306233Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5331713Z Entering 'third_party/sleef'
2022-09-27T16:20:11.5351344Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5377000Z Entering 'third_party/tbb'
2022-09-27T16:20:11.5397218Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5425490Z Entering 'third_party/tensorpipe'
2022-09-27T16:20:11.5444444Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5471116Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:20:11.5489284Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5515532Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:20:11.5534303Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5559997Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:20:11.5578474Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5604431Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:20:11.5623455Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5649141Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:20:11.5667981Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5696252Z Entering 'third_party/zstd'
2022-09-27T16:20:11.5715355Z http.https://github.com/.extraheader
2022-09-27T16:20:11.5941743Z Cleaning up orphan processes