2022-09-27T15:36:22.9983055Z Requested labels: linux.8xlarge.nvidia.gpu
2022-09-27T15:36:22.9983125Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/85462/merge
2022-09-27T15:36:22.9983148Z Waiting for a runner to pick up this job...
2022-09-27T15:36:40.6208404Z Job is about to start running on the runner: i-00fe60cd3c4c7e69b (organization)
2022-09-27T15:36:45.2719299Z Current runner version: '2.296.2'
2022-09-27T15:36:45.2726943Z Runner name: 'i-00fe60cd3c4c7e69b'
2022-09-27T15:36:45.2727617Z Runner group name: 'Default'
2022-09-27T15:36:45.2728506Z Machine name: 'ip-10-0-0-135'
2022-09-27T15:36:45.2731586Z ##[group]GITHUB_TOKEN Permissions
2022-09-27T15:36:45.2732605Z Actions: read
2022-09-27T15:36:45.2733001Z Checks: read
2022-09-27T15:36:45.2733452Z Contents: read
2022-09-27T15:36:45.2733903Z Deployments: read
2022-09-27T15:36:45.2734299Z Discussions: read
2022-09-27T15:36:45.2734789Z Issues: read
2022-09-27T15:36:45.2735231Z Metadata: read
2022-09-27T15:36:45.2735646Z Packages: read
2022-09-27T15:36:45.2736078Z Pages: read
2022-09-27T15:36:45.2736565Z PullRequests: read
2022-09-27T15:36:45.2737020Z RepositoryProjects: read
2022-09-27T15:36:45.2737505Z SecurityEvents: read
2022-09-27T15:36:45.2737957Z Statuses: read
2022-09-27T15:36:45.2738337Z ##[endgroup]
2022-09-27T15:36:45.2742663Z Secret source: None
2022-09-27T15:36:45.2743493Z Prepare workflow directory
2022-09-27T15:36:45.4043348Z Prepare all required actions
2022-09-27T15:36:45.4268054Z Getting action download info
2022-09-27T15:36:45.6307936Z Download action repository 'pytorch/pytorch@master' (SHA:15c52ffc4f9a02f7078033677d44ccd760107952)
2022-09-27T15:36:48.9189717Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-09-27T15:36:49.0498229Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886)
2022-09-27T15:36:49.3522285Z Getting action download info
2022-09-27T15:36:49.5318626Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-09-27T15:36:49.7997887Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-09-27T15:36:49.8000470Z ##[group] Inputs
2022-09-27T15:36:49.8000850Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:36:49.8002028Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-09-27T15:36:49.8003298Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:36:49.8003787Z   sync-tag: 
2022-09-27T15:36:49.8004036Z ##[endgroup]
2022-09-27T15:36:49.8004871Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu)
2022-09-27T15:36:49.9131939Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-09-27T15:36:49.9132371Z with:
2022-09-27T15:36:49.9132636Z   submodules: recursive
2022-09-27T15:36:49.9132920Z   fetch-depth: 0
2022-09-27T15:36:49.9133146Z env:
2022-09-27T15:36:49.9133400Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:49.9133676Z ##[endgroup]
2022-09-27T15:36:49.9440247Z ##[group]Run retry () {
2022-09-27T15:36:49.9440586Z retry () {
2022-09-27T15:36:49.9440911Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-09-27T15:36:49.9441199Z }
2022-09-27T15:36:49.9441472Z echo "${GITHUB_WORKSPACE}"
2022-09-27T15:36:49.9441790Z if [ -z "${NO_SUDO}" ]; then
2022-09-27T15:36:49.9442097Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:36:49.9442392Z else
2022-09-27T15:36:49.9442682Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:36:49.9443137Z fi
2022-09-27T15:36:49.9443488Z mkdir "${GITHUB_WORKSPACE}"
2022-09-27T15:36:49.9461835Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:36:49.9462188Z env:
2022-09-27T15:36:49.9462453Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:49.9462706Z   NO_SUDO: 
2022-09-27T15:36:49.9462959Z ##[endgroup]
2022-09-27T15:36:49.9694544Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:50.0320200Z ##[group]Run malfet/checkout@silent-checkout
2022-09-27T15:36:50.0320524Z with:
2022-09-27T15:36:50.0320803Z   ref: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:36:50.0321096Z   fetch-depth: 0
2022-09-27T15:36:50.0321338Z   submodules: recursive
2022-09-27T15:36:50.0321611Z   quiet-checkout: true
2022-09-27T15:36:50.0321902Z   repository: pytorch/pytorch
2022-09-27T15:36:50.0322344Z   token: ***
2022-09-27T15:36:50.0322619Z   ssh-strict: true
2022-09-27T15:36:50.0323062Z   persist-credentials: true
2022-09-27T15:36:50.0323360Z   clean: true
2022-09-27T15:36:50.0323588Z   lfs: false
2022-09-27T15:36:50.0323862Z   set-safe-directory: true
2022-09-27T15:36:50.0324121Z env:
2022-09-27T15:36:50.0324346Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:50.0324617Z ##[endgroup]
2022-09-27T15:36:50.1879362Z Syncing repository: pytorch/pytorch
2022-09-27T15:36:50.1881307Z ##[group]Getting Git version info
2022-09-27T15:36:50.1881878Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:36:50.1882477Z [command]/usr/bin/git version
2022-09-27T15:36:50.1882769Z git version 2.37.1
2022-09-27T15:36:50.1890374Z ##[endgroup]
2022-09-27T15:36:50.1912605Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/87f10749-2340-4fc3-90bf-da0a70f51788' before making global git config changes
2022-09-27T15:36:50.1913198Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T15:36:50.1921232Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:50.1966093Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:36:50.1972077Z ##[group]Initializing the repository
2022-09-27T15:36:50.1978464Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:50.2011323Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-09-27T15:36:50.2011946Z hint: is subject to change. To configure the initial branch name to use in all
2022-09-27T15:36:50.2012557Z hint: of your new repositories, which will suppress this warning, call:
2022-09-27T15:36:50.2012896Z hint: 
2022-09-27T15:36:50.2013255Z hint: 	git config --global init.defaultBranch <name>
2022-09-27T15:36:50.2013566Z hint: 
2022-09-27T15:36:50.2013969Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-09-27T15:36:50.2014465Z hint: 'development'. The just-created branch can be renamed via this command:
2022-09-27T15:36:50.2014840Z hint: 
2022-09-27T15:36:50.2015353Z hint: 	git branch -m <name>
2022-09-27T15:36:50.2015829Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-09-27T15:36:50.2027203Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-09-27T15:36:50.2062928Z ##[endgroup]
2022-09-27T15:36:50.2063484Z ##[group]Disabling automatic garbage collection
2022-09-27T15:36:50.2068966Z [command]/usr/bin/git config --local gc.auto 0
2022-09-27T15:36:50.2102431Z ##[endgroup]
2022-09-27T15:36:50.2102956Z ##[group]Setting up auth
2022-09-27T15:36:50.2112941Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T15:36:50.2150776Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T15:36:50.2486479Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T15:36:50.2521316Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T15:36:50.2824888Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:36:50.2872977Z ##[endgroup]
2022-09-27T15:36:50.2873500Z ##[group]Fetching the repository
2022-09-27T15:36:50.2881863Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-09-27T15:37:39.1118559Z [command]/usr/bin/git rev-parse --verify --quiet 52424e2bf38e454d535881fed9628d3e20f4f944^{object}
2022-09-27T15:37:39.1159939Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:37:40.2171628Z ##[endgroup]
2022-09-27T15:37:40.2172317Z ##[group]Determining the checkout info
2022-09-27T15:37:40.2175422Z ##[endgroup]
2022-09-27T15:37:40.2176168Z ##[group]Checking out the ref
2022-09-27T15:37:40.2182597Z [command]/usr/bin/git checkout --quiet --force 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:37:41.9172592Z ##[endgroup]
2022-09-27T15:37:41.9173053Z ##[group]Setting up auth for fetching submodules
2022-09-27T15:37:41.9181500Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:37:41.9241238Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-09-27T15:37:41.9276983Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-09-27T15:37:41.9312134Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-09-27T15:37:41.9343418Z ##[endgroup]
2022-09-27T15:37:41.9343804Z ##[group]Fetching submodules
2022-09-27T15:37:41.9349031Z [command]/usr/bin/git submodule sync --recursive
2022-09-27T15:37:41.9679622Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-09-27T15:37:41.9991963Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-09-27T15:37:41.9994308Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-09-27T15:37:41.9997186Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-09-27T15:37:42.0000410Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-09-27T15:37:42.0003708Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-09-27T15:37:42.0007716Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-09-27T15:37:42.0010930Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-09-27T15:37:42.0015108Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-09-27T15:37:42.0019007Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-09-27T15:37:42.0023108Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-09-27T15:37:42.0027125Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-09-27T15:37:42.0031355Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-09-27T15:37:42.0035993Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-09-27T15:37:42.0040177Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-09-27T15:37:42.0045052Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-09-27T15:37:42.0049797Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-09-27T15:37:42.0055231Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-09-27T15:37:42.0060187Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:37:42.0065350Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-09-27T15:37:42.0070519Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-09-27T15:37:42.0076113Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-09-27T15:37:42.0081770Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-09-27T15:37:42.0087367Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-09-27T15:37:42.0093872Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-09-27T15:37:42.0099690Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-09-27T15:37:42.0105766Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-09-27T15:37:42.0111968Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-09-27T15:37:42.0118188Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-09-27T15:37:42.0124693Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-09-27T15:37:42.0131806Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-09-27T15:37:42.0138739Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-09-27T15:37:42.0145524Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-09-27T15:37:42.0152603Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-09-27T15:37:42.0159751Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-09-27T15:37:42.0166847Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-09-27T15:37:42.0174540Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-09-27T15:37:42.0181886Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-09-27T15:37:42.0189372Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-09-27T15:37:42.0197097Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-09-27T15:37:42.0205222Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-09-27T15:37:42.0213495Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-09-27T15:37:42.0242709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-09-27T15:37:42.3560247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-09-27T15:37:42.6063483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-09-27T15:37:42.8330351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-09-27T15:37:43.1521913Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-09-27T15:37:43.4937342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-09-27T15:37:45.6949882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-09-27T15:37:50.6953961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-09-27T15:37:51.1155248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-09-27T15:37:51.8621669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-09-27T15:37:53.3382380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-09-27T15:37:54.7794408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-09-27T15:37:56.2291483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-09-27T15:38:03.0585595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-09-27T15:38:03.8288545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-09-27T15:38:05.1012655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-09-27T15:38:06.2877702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-09-27T15:38:06.5306621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-09-27T15:38:07.1050326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-09-27T15:38:07.4732991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-09-27T15:38:08.6537240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-09-27T15:38:09.0802302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-09-27T15:38:09.2966463Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-09-27T15:38:09.5853364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-09-27T15:38:11.1951613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-09-27T15:38:11.6985322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-09-27T15:38:12.1178087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-09-27T15:38:18.3051710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-09-27T15:38:19.7824507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-09-27T15:38:20.2499652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-09-27T15:38:20.5018196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-09-27T15:38:26.5333302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-09-27T15:38:26.7488903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-09-27T15:38:26.9801357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-09-27T15:38:27.8311550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-09-27T15:38:28.0660192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-09-27T15:38:28.4077733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-09-27T15:38:28.7264287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-09-27T15:38:29.2928413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-09-27T15:38:31.7698079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-09-27T15:38:32.2875496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-09-27T15:38:34.5983324Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-09-27T15:38:34.6108996Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-09-27T15:38:34.6204208Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-09-27T15:38:34.6489983Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-09-27T15:38:34.6765028Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-09-27T15:38:34.7248018Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-09-27T15:38:35.5254321Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-09-27T15:38:35.5506524Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:38:35.6743219Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-09-27T15:38:35.7149346Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-09-27T15:38:36.0798179Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-09-27T15:38:36.6016742Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-09-27T15:38:36.9109249Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-09-27T15:38:36.9671346Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030'
2022-09-27T15:38:36.9689196Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:38:36.9692996Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:38:36.9696194Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:38:36.9699853Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:38:36.9727055Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-09-27T15:38:37.8146199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-09-27T15:38:38.5898787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-09-27T15:38:39.5811393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-09-27T15:38:39.9126666Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-09-27T15:38:40.0377201Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-09-27T15:38:40.1072569Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-09-27T15:38:40.1183271Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-09-27T15:38:40.2252155Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-09-27T15:38:40.2646420Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-09-27T15:38:40.2747026Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-09-27T15:38:40.3217916Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-09-27T15:38:40.3505790Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-09-27T15:38:40.4057565Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-09-27T15:38:40.4189396Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-09-27T15:38:40.4206234Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-09-27T15:38:40.4233992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-09-27T15:38:47.6856728Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-09-27T15:38:47.6876061Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:38:47.6904172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-09-27T15:38:54.9746344Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-09-27T15:38:54.9869454Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-09-27T15:38:55.0039527Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-09-27T15:38:55.1194997Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-09-27T15:38:55.1213733Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:38:55.1217051Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:38:55.1245627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-09-27T15:38:56.3824622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-09-27T15:38:57.3899700Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-09-27T15:38:57.4548737Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-09-27T15:38:57.4788455Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-09-27T15:38:57.4941262Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-09-27T15:38:57.6293719Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-09-27T15:38:57.9582459Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-09-27T15:38:57.9615069Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-09-27T15:38:57.9618264Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-09-27T15:38:57.9646168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-09-27T15:38:58.7953009Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-09-27T15:38:59.6442885Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:38:59.6823633Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-09-27T15:38:59.7006415Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-09-27T15:38:59.7024023Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:38:59.7050245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-09-27T15:39:01.3670750Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-09-27T15:39:01.3694032Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:01.3697132Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:01.3725707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-09-27T15:39:01.9121557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-09-27T15:39:02.8563240Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-09-27T15:39:02.9331998Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-09-27T15:39:02.9349002Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:02.9376504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-09-27T15:39:03.2498440Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:39:03.2605192Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-09-27T15:39:03.5832665Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-09-27T15:39:03.5855730Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:03.5858785Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:03.5887786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-09-27T15:39:04.2689545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-09-27T15:39:05.3129047Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-09-27T15:39:05.3962936Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-09-27T15:39:05.4059530Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-09-27T15:39:05.4187024Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-09-27T15:39:05.4607753Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-09-27T15:39:05.4713755Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-09-27T15:39:05.5068853Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-09-27T15:39:05.5178953Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-09-27T15:39:05.5722824Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-09-27T15:39:05.7246374Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-09-27T15:39:05.7580689Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-09-27T15:39:05.7598582Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:05.7601601Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:05.7604808Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:05.7608089Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:05.7635413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-09-27T15:39:07.5217529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-09-27T15:39:07.8210363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-09-27T15:39:09.0921232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-09-27T15:39:09.9857652Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-09-27T15:39:10.0025239Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-09-27T15:39:10.0811209Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-09-27T15:39:10.1130734Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-09-27T15:39:10.1147432Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:10.1175407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-09-27T15:39:10.3620187Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:39:10.5224783Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-09-27T15:39:10.5261974Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-09-27T15:39:10.5594642Z Entering 'android/libs/fbjni'
2022-09-27T15:39:10.5638062Z Entering 'third_party/FP16'
2022-09-27T15:39:10.5681748Z Entering 'third_party/FXdiv'
2022-09-27T15:39:10.5724657Z Entering 'third_party/NNPACK'
2022-09-27T15:39:10.5770349Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:10.5814602Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:10.5860171Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:10.5914560Z Entering 'third_party/benchmark'
2022-09-27T15:39:10.5958101Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:10.6002766Z Entering 'third_party/cub'
2022-09-27T15:39:10.6048379Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:10.6100566Z Entering 'third_party/cutlass'
2022-09-27T15:39:10.6152552Z Entering 'third_party/eigen'
2022-09-27T15:39:10.6198138Z Entering 'third_party/fbgemm'
2022-09-27T15:39:10.6242138Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:10.6286055Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:10.6329703Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:10.6373186Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:10.6415619Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:10.6460170Z Entering 'third_party/fmt'
2022-09-27T15:39:10.6503421Z Entering 'third_party/foxi'
2022-09-27T15:39:10.6545829Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:10.6588316Z Entering 'third_party/gloo'
2022-09-27T15:39:10.6631658Z Entering 'third_party/googletest'
2022-09-27T15:39:10.6675407Z Entering 'third_party/ideep'
2022-09-27T15:39:10.6717798Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:10.6762833Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:10.6813342Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:10.6856927Z Entering 'third_party/ittapi'
2022-09-27T15:39:10.6899354Z Entering 'third_party/kineto'
2022-09-27T15:39:10.6941483Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:10.6984177Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:10.7028220Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:10.7072563Z Entering 'third_party/neon2sse'
2022-09-27T15:39:10.7114456Z Entering 'third_party/nlohmann'
2022-09-27T15:39:10.7158525Z Entering 'third_party/onnx'
2022-09-27T15:39:10.7216160Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:10.7259159Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:10.7305565Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:10.7349093Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:10.7398076Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:10.7441497Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:10.7483072Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:10.7530633Z Entering 'third_party/pocketfft'
2022-09-27T15:39:10.7573263Z Entering 'third_party/protobuf'
2022-09-27T15:39:10.7620126Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:10.7662790Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:10.7708272Z Entering 'third_party/psimd'
2022-09-27T15:39:10.7750694Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:10.7792974Z Entering 'third_party/pybind11'
2022-09-27T15:39:10.7835572Z Entering 'third_party/python-enum'
2022-09-27T15:39:10.7877511Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:10.7920243Z Entering 'third_party/python-six'
2022-09-27T15:39:10.7962521Z Entering 'third_party/sleef'
2022-09-27T15:39:10.8005635Z Entering 'third_party/tbb'
2022-09-27T15:39:10.8051077Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:10.8094037Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:10.8136445Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:10.8177806Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:10.8219447Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:10.8261050Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:10.8305738Z Entering 'third_party/zstd'
2022-09-27T15:39:10.8360413Z ##[endgroup]
2022-09-27T15:39:10.8363730Z ##[group]Persisting credentials for submodules
2022-09-27T15:39:10.8371396Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-09-27T15:39:10.8690775Z Entering 'android/libs/fbjni'
2022-09-27T15:39:10.8731519Z Entering 'third_party/FP16'
2022-09-27T15:39:10.8774842Z Entering 'third_party/FXdiv'
2022-09-27T15:39:10.8817531Z Entering 'third_party/NNPACK'
2022-09-27T15:39:10.8859067Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:10.8900597Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:10.8942436Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:10.8995413Z Entering 'third_party/benchmark'
2022-09-27T15:39:10.9038551Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:10.9081870Z Entering 'third_party/cub'
2022-09-27T15:39:10.9123525Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:10.9171410Z Entering 'third_party/cutlass'
2022-09-27T15:39:10.9219524Z Entering 'third_party/eigen'
2022-09-27T15:39:10.9263697Z Entering 'third_party/fbgemm'
2022-09-27T15:39:10.9304772Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:10.9347536Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:10.9390173Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:10.9432463Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:10.9474713Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:10.9517881Z Entering 'third_party/fmt'
2022-09-27T15:39:10.9559603Z Entering 'third_party/foxi'
2022-09-27T15:39:10.9601215Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:10.9643402Z Entering 'third_party/gloo'
2022-09-27T15:39:10.9686370Z Entering 'third_party/googletest'
2022-09-27T15:39:10.9728225Z Entering 'third_party/ideep'
2022-09-27T15:39:10.9769685Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:10.9814513Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:10.9863026Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:10.9905515Z Entering 'third_party/ittapi'
2022-09-27T15:39:10.9946452Z Entering 'third_party/kineto'
2022-09-27T15:39:10.9988045Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:11.0030181Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:11.0074412Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:11.0115864Z Entering 'third_party/neon2sse'
2022-09-27T15:39:11.0157042Z Entering 'third_party/nlohmann'
2022-09-27T15:39:11.0200346Z Entering 'third_party/onnx'
2022-09-27T15:39:11.0254498Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.0296422Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.0340793Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:11.0381220Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:11.0427723Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.0471457Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.0512767Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.0559226Z Entering 'third_party/pocketfft'
2022-09-27T15:39:11.0601428Z Entering 'third_party/protobuf'
2022-09-27T15:39:11.0647108Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:11.0687833Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:11.0733082Z Entering 'third_party/psimd'
2022-09-27T15:39:11.0774945Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:11.0817874Z Entering 'third_party/pybind11'
2022-09-27T15:39:11.0858873Z Entering 'third_party/python-enum'
2022-09-27T15:39:11.0900200Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:11.0941344Z Entering 'third_party/python-six'
2022-09-27T15:39:11.0982235Z Entering 'third_party/sleef'
2022-09-27T15:39:11.1023661Z Entering 'third_party/tbb'
2022-09-27T15:39:11.1066553Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:11.1108154Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:11.1149303Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:11.1190938Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:11.1232157Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:11.1272604Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.1316025Z Entering 'third_party/zstd'
2022-09-27T15:39:11.1372780Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-09-27T15:39:11.1684215Z Entering 'android/libs/fbjni'
2022-09-27T15:39:11.1723520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-09-27T15:39:11.1740977Z Entering 'third_party/FP16'
2022-09-27T15:39:11.1779969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-09-27T15:39:11.1797008Z Entering 'third_party/FXdiv'
2022-09-27T15:39:11.1836117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-09-27T15:39:11.1853233Z Entering 'third_party/NNPACK'
2022-09-27T15:39:11.1891897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-09-27T15:39:11.1908947Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:11.1947173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-09-27T15:39:11.1964319Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:11.2004368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-09-27T15:39:11.2022406Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:11.2061173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-09-27T15:39:11.2089030Z Entering 'third_party/benchmark'
2022-09-27T15:39:11.2127488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:11.2144881Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:11.2182940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:39:11.2200704Z Entering 'third_party/cub'
2022-09-27T15:39:11.2239951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-09-27T15:39:11.2259193Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:11.2297696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-09-27T15:39:11.2320631Z Entering 'third_party/cutlass'
2022-09-27T15:39:11.2358890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-09-27T15:39:11.2382901Z Entering 'third_party/eigen'
2022-09-27T15:39:11.2421776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-09-27T15:39:11.2441368Z Entering 'third_party/fbgemm'
2022-09-27T15:39:11.2481840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-09-27T15:39:11.2500471Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:11.2540196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-09-27T15:39:11.2557117Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:11.2595424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:39:11.2613484Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:11.2652908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:11.2670471Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:11.2709217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-09-27T15:39:11.2727448Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:11.2766036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-09-27T15:39:11.2785544Z Entering 'third_party/fmt'
2022-09-27T15:39:11.2823897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-09-27T15:39:11.2841028Z Entering 'third_party/foxi'
2022-09-27T15:39:11.2879411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-09-27T15:39:11.2897874Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:11.2936318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-09-27T15:39:11.2963931Z Entering 'third_party/gloo'
2022-09-27T15:39:11.2993058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-09-27T15:39:11.3009333Z Entering 'third_party/googletest'
2022-09-27T15:39:11.3048281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:11.3065931Z Entering 'third_party/ideep'
2022-09-27T15:39:11.3105430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-09-27T15:39:11.3121659Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:11.3160223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-09-27T15:39:11.3179888Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:11.3219423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-09-27T15:39:11.3242965Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:11.3282666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-09-27T15:39:11.3300619Z Entering 'third_party/ittapi'
2022-09-27T15:39:11.3339417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-09-27T15:39:11.3355714Z Entering 'third_party/kineto'
2022-09-27T15:39:11.3395377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-09-27T15:39:11.3413446Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:11.3451303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-09-27T15:39:11.3469098Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:11.3508762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:11.3527295Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:11.3566564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-09-27T15:39:11.3584864Z Entering 'third_party/neon2sse'
2022-09-27T15:39:11.3624239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-09-27T15:39:11.3640695Z Entering 'third_party/nlohmann'
2022-09-27T15:39:11.3679637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-09-27T15:39:11.3698981Z Entering 'third_party/onnx'
2022-09-27T15:39:11.3737639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:39:11.3768668Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.3808439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:11.3826497Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.3865508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:11.3884691Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:11.3924628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-09-27T15:39:11.3941782Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:11.3980358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:39:11.4002625Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.4042070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:11.4059879Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.4099269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:11.4116122Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.4155740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:39:11.4178126Z Entering 'third_party/pocketfft'
2022-09-27T15:39:11.4218244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-09-27T15:39:11.4235227Z Entering 'third_party/protobuf'
2022-09-27T15:39:11.4275289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-09-27T15:39:11.4295837Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:11.4334487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:11.4351553Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:11.4390816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:11.4411025Z Entering 'third_party/psimd'
2022-09-27T15:39:11.4450193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-09-27T15:39:11.4467064Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:11.4506896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-09-27T15:39:11.4523753Z Entering 'third_party/pybind11'
2022-09-27T15:39:11.4562858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:11.4581021Z Entering 'third_party/python-enum'
2022-09-27T15:39:11.4619048Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-09-27T15:39:11.4636310Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:11.4675944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-09-27T15:39:11.4694308Z Entering 'third_party/python-six'
2022-09-27T15:39:11.4732782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-09-27T15:39:11.4749563Z Entering 'third_party/sleef'
2022-09-27T15:39:11.4788501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-09-27T15:39:11.4805610Z Entering 'third_party/tbb'
2022-09-27T15:39:11.4844236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-09-27T15:39:11.4865399Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:11.4903681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-09-27T15:39:11.4920874Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:11.4959122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:11.4976780Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:11.5014775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-09-27T15:39:11.5031697Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:11.5070360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-09-27T15:39:11.5087931Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:11.5126077Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:11.5142721Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.5181911Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:39:11.5202046Z Entering 'third_party/zstd'
2022-09-27T15:39:11.5240428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-09-27T15:39:11.6127550Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-09-27T15:39:11.6444676Z Entering 'android/libs/fbjni'
2022-09-27T15:39:11.6488374Z Entering 'third_party/FP16'
2022-09-27T15:39:11.6531741Z Entering 'third_party/FXdiv'
2022-09-27T15:39:11.6575482Z Entering 'third_party/NNPACK'
2022-09-27T15:39:11.6618294Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:11.6662769Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:11.6706040Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:11.6761313Z Entering 'third_party/benchmark'
2022-09-27T15:39:11.6804411Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:11.6847164Z Entering 'third_party/cub'
2022-09-27T15:39:11.6891288Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:11.6939734Z Entering 'third_party/cutlass'
2022-09-27T15:39:11.6988142Z Entering 'third_party/eigen'
2022-09-27T15:39:11.7033815Z Entering 'third_party/fbgemm'
2022-09-27T15:39:11.7077866Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:11.7120321Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:11.7162574Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:11.7205459Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:11.7248320Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:11.7294532Z Entering 'third_party/fmt'
2022-09-27T15:39:11.7337460Z Entering 'third_party/foxi'
2022-09-27T15:39:11.7380256Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:11.7422936Z Entering 'third_party/gloo'
2022-09-27T15:39:11.7469090Z Entering 'third_party/googletest'
2022-09-27T15:39:11.7513057Z Entering 'third_party/ideep'
2022-09-27T15:39:11.7557000Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:11.7602189Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:11.7653339Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:11.7696107Z Entering 'third_party/ittapi'
2022-09-27T15:39:11.7739346Z Entering 'third_party/kineto'
2022-09-27T15:39:11.7783074Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:11.7826343Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:11.7870567Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:11.7914443Z Entering 'third_party/neon2sse'
2022-09-27T15:39:11.7956733Z Entering 'third_party/nlohmann'
2022-09-27T15:39:11.8001321Z Entering 'third_party/onnx'
2022-09-27T15:39:11.8057368Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.8100191Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.8144400Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:11.8186846Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:11.8234033Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:11.8276898Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:11.8321307Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.8368801Z Entering 'third_party/pocketfft'
2022-09-27T15:39:11.8412197Z Entering 'third_party/protobuf'
2022-09-27T15:39:11.8459554Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:11.8504763Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:11.8549308Z Entering 'third_party/psimd'
2022-09-27T15:39:11.8591839Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:11.8635575Z Entering 'third_party/pybind11'
2022-09-27T15:39:11.8677769Z Entering 'third_party/python-enum'
2022-09-27T15:39:11.8720857Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:11.8765007Z Entering 'third_party/python-six'
2022-09-27T15:39:11.8808908Z Entering 'third_party/sleef'
2022-09-27T15:39:11.8853201Z Entering 'third_party/tbb'
2022-09-27T15:39:11.8900198Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:11.8945018Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:11.8988869Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:11.9033411Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:11.9076850Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:11.9119390Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:11.9166289Z Entering 'third_party/zstd'
2022-09-27T15:39:11.9227557Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-09-27T15:39:11.9548496Z Entering 'android/libs/fbjni'
2022-09-27T15:39:11.9592825Z Entering 'third_party/FP16'
2022-09-27T15:39:11.9636432Z Entering 'third_party/FXdiv'
2022-09-27T15:39:11.9679533Z Entering 'third_party/NNPACK'
2022-09-27T15:39:11.9723923Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:11.9766709Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:11.9809915Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:11.9866037Z Entering 'third_party/benchmark'
2022-09-27T15:39:11.9908716Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:11.9953203Z Entering 'third_party/cub'
2022-09-27T15:39:11.9998360Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:12.0048714Z Entering 'third_party/cutlass'
2022-09-27T15:39:12.0100624Z Entering 'third_party/eigen'
2022-09-27T15:39:12.0147708Z Entering 'third_party/fbgemm'
2022-09-27T15:39:12.0192255Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:12.0236575Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:12.0279196Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:12.0321943Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:12.0365579Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:12.0411778Z Entering 'third_party/fmt'
2022-09-27T15:39:12.0454297Z Entering 'third_party/foxi'
2022-09-27T15:39:12.0497765Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:12.0540991Z Entering 'third_party/gloo'
2022-09-27T15:39:12.0585122Z Entering 'third_party/googletest'
2022-09-27T15:39:12.0629846Z Entering 'third_party/ideep'
2022-09-27T15:39:12.0672954Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:12.0718301Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:12.0768292Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:12.0813137Z Entering 'third_party/ittapi'
2022-09-27T15:39:12.0855761Z Entering 'third_party/kineto'
2022-09-27T15:39:12.0898569Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:12.0941732Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:12.0986164Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:12.1029514Z Entering 'third_party/neon2sse'
2022-09-27T15:39:12.1072345Z Entering 'third_party/nlohmann'
2022-09-27T15:39:12.1117219Z Entering 'third_party/onnx'
2022-09-27T15:39:12.1175166Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:12.1219875Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:12.1264909Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:12.1308200Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:12.1357022Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:12.1400820Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:12.1442886Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:12.1491993Z Entering 'third_party/pocketfft'
2022-09-27T15:39:12.1533918Z Entering 'third_party/protobuf'
2022-09-27T15:39:12.1580976Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:12.1623348Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:12.1668802Z Entering 'third_party/psimd'
2022-09-27T15:39:12.1712374Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:12.1757374Z Entering 'third_party/pybind11'
2022-09-27T15:39:12.1802442Z Entering 'third_party/python-enum'
2022-09-27T15:39:12.1845897Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:12.1888517Z Entering 'third_party/python-six'
2022-09-27T15:39:12.1932897Z Entering 'third_party/sleef'
2022-09-27T15:39:12.1975901Z Entering 'third_party/tbb'
2022-09-27T15:39:12.2021438Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:12.2065747Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:12.2109663Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:12.2153088Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:12.2196349Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:12.2238074Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:12.2284382Z Entering 'third_party/zstd'
2022-09-27T15:39:12.2340682Z ##[endgroup]
2022-09-27T15:39:12.2388869Z [command]/usr/bin/git log -1 --format='%H'
2022-09-27T15:39:12.2418864Z '52424e2bf38e454d535881fed9628d3e20f4f944'
2022-09-27T15:39:12.2577351Z Prepare all required actions
2022-09-27T15:39:12.2659423Z ##[group]Run ./.github/actions/setup-linux
2022-09-27T15:39:12.2659710Z env:
2022-09-27T15:39:12.2659962Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:12.2660203Z ##[endgroup]
2022-09-27T15:39:12.2680294Z ##[group]Run set -euo pipefail
2022-09-27T15:39:12.2680620Z set -euo pipefail
2022-09-27T15:39:12.2680911Z function get_ec2_metadata() {
2022-09-27T15:39:12.2681233Z   # Pulled from instance metadata endpoint for EC2
2022-09-27T15:39:12.2681719Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-09-27T15:39:12.2682129Z   category=$1
2022-09-27T15:39:12.2682462Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-09-27T15:39:12.2682751Z }
2022-09-27T15:39:12.2683203Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-09-27T15:39:12.2683601Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-09-27T15:39:12.2683965Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-09-27T15:39:12.2684309Z echo "system info $(uname -a)"
2022-09-27T15:39:12.2697756Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:12.2698037Z env:
2022-09-27T15:39:12.2698283Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:12.2698545Z ##[endgroup]
2022-09-27T15:39:12.2800646Z ami-id: ami-096198a0bccc6bad4
2022-09-27T15:39:12.2866345Z instance-id: i-00fe60cd3c4c7e69b
2022-09-27T15:39:12.2927280Z instance-type: g3.8xlarge
2022-09-27T15:39:12.2936745Z system info Linux ip-10-0-0-135.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-09-27T15:39:12.2957243Z ##[group]Run if systemctl is-active --quiet docker; then
2022-09-27T15:39:12.2957618Z if systemctl is-active --quiet docker; then
2022-09-27T15:39:12.2957957Z     echo "Docker daemon is running...";
2022-09-27T15:39:12.2958219Z else
2022-09-27T15:39:12.2958538Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-09-27T15:39:12.2958850Z fi
2022-09-27T15:39:12.2970265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:12.2970571Z env:
2022-09-27T15:39:12.2970822Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:12.2971058Z ##[endgroup]
2022-09-27T15:39:12.3022657Z Docker daemon is running...
2022-09-27T15:39:12.3043306Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:39:12.3043781Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:39:12.3044169Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:12.3044737Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-09-27T15:39:12.3045209Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-09-27T15:39:12.3057152Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:12.3057442Z env:
2022-09-27T15:39:12.3057697Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:12.3057978Z   AWS_RETRY_MODE: standard
2022-09-27T15:39:12.3058220Z   AWS_MAX_ATTEMPTS: 5
2022-09-27T15:39:12.3058496Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T15:39:12.3058768Z ##[endgroup]
2022-09-27T15:39:13.2913805Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-09-27T15:39:13.2914253Z Configure a credential helper to remove this warning. See
2022-09-27T15:39:13.2914805Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-09-27T15:39:13.2915082Z 
2022-09-27T15:39:13.2915727Z Login Succeeded
2022-09-27T15:39:13.2962047Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:13.2962491Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:13.2962998Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:13.2976204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:13.2976516Z env:
2022-09-27T15:39:13.2976772Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:13.2977023Z ##[endgroup]
2022-09-27T15:39:13.3054410Z Prepare all required actions
2022-09-27T15:39:13.3054790Z Getting action download info
2022-09-27T15:39:13.4539569Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638)
2022-09-27T15:39:13.6076605Z ##[group]Run ./.github/actions/setup-ssh
2022-09-27T15:39:13.6076890Z with:
2022-09-27T15:39:13.6077365Z   github-secret: ***
2022-09-27T15:39:13.6077621Z env:
2022-09-27T15:39:13.6077869Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:13.6078110Z ##[endgroup]
2022-09-27T15:39:13.6105681Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-09-27T15:39:13.6106106Z with:
2022-09-27T15:39:13.6106497Z   GITHUB_TOKEN: ***
2022-09-27T15:39:13.6106756Z   activate-with-label: false
2022-09-27T15:39:13.6107025Z   label: with-ssh
2022-09-27T15:39:13.6107296Z   remove-existing-keys: true
2022-09-27T15:39:13.6107533Z env:
2022-09-27T15:39:13.6107776Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:13.6108035Z ##[endgroup]
2022-09-27T15:39:13.9937440Z Grabbing public ssh keys from https://github.com/kongzii.keys
2022-09-27T15:39:14.0789806Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-09-27T15:39:14.0810963Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-09-27T15:39:14.0847862Z Login using: ssh ec2-user@ec2-54-166-137-31.compute-1.amazonaws.com
2022-09-27T15:39:14.0911490Z Prepare all required actions
2022-09-27T15:39:14.0937359Z ##[group]Run ./.github/actions/pull-docker-image
2022-09-27T15:39:14.0937658Z with:
2022-09-27T15:39:14.0938182Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:39:14.0938641Z env:
2022-09-27T15:39:14.0938887Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:14.0939156Z ##[endgroup]
2022-09-27T15:39:14.1067288Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:14.1067677Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:14.1068051Z # ignore output since only exit code is used for conditional
2022-09-27T15:39:14.1068421Z # only pull docker image if it's not available locally
2022-09-27T15:39:14.1068834Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-09-27T15:39:14.1069262Z   retry docker pull "${DOCKER_IMAGE}"
2022-09-27T15:39:14.1069526Z fi
2022-09-27T15:39:14.1082269Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:14.1082576Z env:
2022-09-27T15:39:14.1082831Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:14.1083338Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:39:14.1083829Z ##[endgroup]
2022-09-27T15:39:14.3686385Z e66cf5fa0a4d4ed512901b12ccdab95cca946a29: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-09-27T15:39:14.3687348Z 40dd5be53814: Pulling fs layer
2022-09-27T15:39:14.3687903Z bd44602516a4: Pulling fs layer
2022-09-27T15:39:14.3688342Z 8ebfb31ea67d: Pulling fs layer
2022-09-27T15:39:14.3688989Z 1589dc294916: Pulling fs layer
2022-09-27T15:39:14.3689493Z 2c3a764ff1ef: Pulling fs layer
2022-09-27T15:39:14.3690405Z 2fb24fb5f7cb: Pulling fs layer
2022-09-27T15:39:14.3691017Z d6e4b45751c9: Pulling fs layer
2022-09-27T15:39:14.3691560Z 98a26bc0781e: Pulling fs layer
2022-09-27T15:39:14.3692109Z 07c42b0591b2: Pulling fs layer
2022-09-27T15:39:14.3692624Z 9be88323b57e: Pulling fs layer
2022-09-27T15:39:14.3693197Z 2c7b68ade49f: Pulling fs layer
2022-09-27T15:39:14.3693695Z 44206692de1d: Pulling fs layer
2022-09-27T15:39:14.3694087Z 07c42b0591b2: Waiting
2022-09-27T15:39:14.3694467Z f751461554fa: Pulling fs layer
2022-09-27T15:39:14.3695057Z 316750fef2e6: Pulling fs layer
2022-09-27T15:39:14.3695622Z 2fb24fb5f7cb: Waiting
2022-09-27T15:39:14.3696094Z c069021d810b: Pulling fs layer
2022-09-27T15:39:14.3696638Z e0fdd58e805b: Pulling fs layer
2022-09-27T15:39:14.3697222Z 751286b45698: Pulling fs layer
2022-09-27T15:39:14.3697680Z 0c8bd29be614: Pulling fs layer
2022-09-27T15:39:14.3697955Z 3bb9e7ea569e: Pulling fs layer
2022-09-27T15:39:14.3698220Z 2c3a764ff1ef: Waiting
2022-09-27T15:39:14.3698449Z 9be88323b57e: Waiting
2022-09-27T15:39:14.3698913Z efeff9c74fbf: Pulling fs layer
2022-09-27T15:39:14.3699515Z 81a5271d43c8: Pulling fs layer
2022-09-27T15:39:14.3700083Z 903ca36d4d71: Pulling fs layer
2022-09-27T15:39:14.3700406Z d52c758f8e75: Pulling fs layer
2022-09-27T15:39:14.3700674Z 316750fef2e6: Waiting
2022-09-27T15:39:14.3701119Z a4ce2fdd9133: Pulling fs layer
2022-09-27T15:39:14.3701362Z 98a26bc0781e: Waiting
2022-09-27T15:39:14.3701628Z cae8823a1cd1: Pulling fs layer
2022-09-27T15:39:14.3701890Z c069021d810b: Waiting
2022-09-27T15:39:14.3702150Z 3298fe919163: Pulling fs layer
2022-09-27T15:39:14.3702387Z d6e4b45751c9: Waiting
2022-09-27T15:39:14.3702650Z b9b9b9d06eef: Pulling fs layer
2022-09-27T15:39:14.3702916Z 3bb9e7ea569e: Waiting
2022-09-27T15:39:14.3703145Z e0fdd58e805b: Waiting
2022-09-27T15:39:14.3703391Z 903ca36d4d71: Waiting
2022-09-27T15:39:14.3703633Z 81a5271d43c8: Waiting
2022-09-27T15:39:14.3703856Z 2c7b68ade49f: Waiting
2022-09-27T15:39:14.3704122Z 62fa99d47769: Pulling fs layer
2022-09-27T15:39:14.3704425Z 17acc9e30503: Pulling fs layer
2022-09-27T15:39:14.3704795Z efeff9c74fbf: Waiting
2022-09-27T15:39:14.3705095Z e8b4222e7a59: Pulling fs layer
2022-09-27T15:39:14.3705500Z b752992950f8: Pulling fs layer
2022-09-27T15:39:14.3705752Z d52c758f8e75: Waiting
2022-09-27T15:39:14.3706006Z cc8443c330a0: Pulling fs layer
2022-09-27T15:39:14.3706303Z 751286b45698: Waiting
2022-09-27T15:39:14.3706535Z c2fcfa2400df: Pulling fs layer
2022-09-27T15:39:14.3706793Z dce607cbd09e: Pulling fs layer
2022-09-27T15:39:14.3707068Z 45b253446018: Pulling fs layer
2022-09-27T15:39:14.3707376Z 752f98c7a6d7: Pulling fs layer
2022-09-27T15:39:14.3707731Z 410e31c94a04: Pulling fs layer
2022-09-27T15:39:14.3707996Z b9b9b9d06eef: Waiting
2022-09-27T15:39:14.3708223Z 45b253446018: Waiting
2022-09-27T15:39:14.3708480Z aa4bb3ec24a7: Pulling fs layer
2022-09-27T15:39:14.3708764Z 6a9eea4b3aa4: Pulling fs layer
2022-09-27T15:39:14.3709006Z cc8443c330a0: Waiting
2022-09-27T15:39:14.3709252Z 752f98c7a6d7: Waiting
2022-09-27T15:39:14.3709510Z a823f5718e87: Pulling fs layer
2022-09-27T15:39:14.3709753Z aa4bb3ec24a7: Waiting
2022-09-27T15:39:14.3710016Z 788ace045743: Pulling fs layer
2022-09-27T15:39:14.3710278Z c2fcfa2400df: Waiting
2022-09-27T15:39:14.3710526Z c76c6ad2ac0f: Pulling fs layer
2022-09-27T15:39:14.3710805Z 6cd5f9a2c4ae: Pulling fs layer
2022-09-27T15:39:14.3711087Z 577da355ab1b: Pulling fs layer
2022-09-27T15:39:14.3711338Z 649c4428b346: Pulling fs layer
2022-09-27T15:39:14.3711623Z eadc05ea2cd3: Pulling fs layer
2022-09-27T15:39:14.3711891Z a4ce2fdd9133: Waiting
2022-09-27T15:39:14.3712225Z 93e5a7080833: Pulling fs layer
2022-09-27T15:39:14.3712548Z e6d72a41a09b: Pulling fs layer
2022-09-27T15:39:14.3712808Z 649c4428b346: Waiting
2022-09-27T15:39:14.3713070Z acb01049a64b: Pulling fs layer
2022-09-27T15:39:14.3713307Z 1589dc294916: Waiting
2022-09-27T15:39:14.3713546Z b752992950f8: Waiting
2022-09-27T15:39:14.3713804Z 343cc73c5973: Pulling fs layer
2022-09-27T15:39:14.3714043Z 6cd5f9a2c4ae: Waiting
2022-09-27T15:39:14.3714443Z 7d69e17e7339: Pulling fs layer
2022-09-27T15:39:14.3714783Z 7d69e17e7339: Waiting
2022-09-27T15:39:14.3714967Z 5d0b32cc6f2a: Pulling fs layer
2022-09-27T15:39:14.3715231Z dce607cbd09e: Waiting
2022-09-27T15:39:14.3715498Z a6c12031bfcf: Pulling fs layer
2022-09-27T15:39:14.3715749Z acb01049a64b: Waiting
2022-09-27T15:39:14.3715991Z 5d0b32cc6f2a: Waiting
2022-09-27T15:39:14.3716240Z a6c12031bfcf: Waiting
2022-09-27T15:39:14.3716459Z a823f5718e87: Waiting
2022-09-27T15:39:14.3716704Z e6d72a41a09b: Waiting
2022-09-27T15:39:14.3716966Z 410e31c94a04: Waiting
2022-09-27T15:39:14.5074352Z bd44602516a4: Verifying Checksum
2022-09-27T15:39:14.5075084Z bd44602516a4: Download complete
2022-09-27T15:39:14.5933593Z 1589dc294916: Verifying Checksum
2022-09-27T15:39:14.5933951Z 1589dc294916: Download complete
2022-09-27T15:39:14.6582600Z 8ebfb31ea67d: Verifying Checksum
2022-09-27T15:39:14.6582982Z 8ebfb31ea67d: Download complete
2022-09-27T15:39:14.6707711Z 2c3a764ff1ef: Download complete
2022-09-27T15:39:14.7002225Z 40dd5be53814: Verifying Checksum
2022-09-27T15:39:14.7002558Z 40dd5be53814: Download complete
2022-09-27T15:39:14.7792462Z d6e4b45751c9: Verifying Checksum
2022-09-27T15:39:14.7793156Z d6e4b45751c9: Download complete
2022-09-27T15:39:14.8520725Z 07c42b0591b2: Download complete
2022-09-27T15:39:14.9334386Z 9be88323b57e: Download complete
2022-09-27T15:39:15.4499446Z 40dd5be53814: Pull complete
2022-09-27T15:39:15.7482946Z bd44602516a4: Pull complete
2022-09-27T15:39:16.2500263Z 8ebfb31ea67d: Pull complete
2022-09-27T15:39:16.3956522Z 1589dc294916: Pull complete
2022-09-27T15:39:16.5139427Z 2c3a764ff1ef: Pull complete
2022-09-27T15:39:20.6744088Z 2c7b68ade49f: Verifying Checksum
2022-09-27T15:39:20.6744489Z 2c7b68ade49f: Download complete
2022-09-27T15:39:20.7479673Z 44206692de1d: Download complete
2022-09-27T15:39:20.8272952Z f751461554fa: Verifying Checksum
2022-09-27T15:39:20.8273274Z f751461554fa: Download complete
2022-09-27T15:39:20.9029249Z 316750fef2e6: Download complete
2022-09-27T15:39:22.3833808Z c069021d810b: Verifying Checksum
2022-09-27T15:39:22.3834477Z c069021d810b: Download complete
2022-09-27T15:39:22.4637491Z e0fdd58e805b: Verifying Checksum
2022-09-27T15:39:22.4637832Z e0fdd58e805b: Download complete
2022-09-27T15:39:22.5826814Z 751286b45698: Verifying Checksum
2022-09-27T15:39:22.5827177Z 751286b45698: Download complete
2022-09-27T15:39:25.8724259Z 2fb24fb5f7cb: Verifying Checksum
2022-09-27T15:39:25.8724666Z 2fb24fb5f7cb: Download complete
2022-09-27T15:39:25.9592535Z 3bb9e7ea569e: Verifying Checksum
2022-09-27T15:39:25.9593011Z 3bb9e7ea569e: Download complete
2022-09-27T15:39:26.0717228Z efeff9c74fbf: Verifying Checksum
2022-09-27T15:39:26.0717644Z efeff9c74fbf: Download complete
2022-09-27T15:39:26.1589624Z 81a5271d43c8: Verifying Checksum
2022-09-27T15:39:26.1590012Z 81a5271d43c8: Download complete
2022-09-27T15:39:26.2709978Z 903ca36d4d71: Verifying Checksum
2022-09-27T15:39:26.2710447Z 903ca36d4d71: Download complete
2022-09-27T15:39:26.3936784Z d52c758f8e75: Verifying Checksum
2022-09-27T15:39:26.3937153Z d52c758f8e75: Download complete
2022-09-27T15:39:26.4717044Z a4ce2fdd9133: Verifying Checksum
2022-09-27T15:39:26.4717389Z a4ce2fdd9133: Download complete
2022-09-27T15:39:28.7079162Z cae8823a1cd1: Verifying Checksum
2022-09-27T15:39:28.7079592Z cae8823a1cd1: Download complete
2022-09-27T15:39:28.7942424Z 3298fe919163: Verifying Checksum
2022-09-27T15:39:28.7943071Z 3298fe919163: Download complete
2022-09-27T15:39:28.8686631Z b9b9b9d06eef: Download complete
2022-09-27T15:39:29.0040326Z 62fa99d47769: Verifying Checksum
2022-09-27T15:39:29.0040738Z 62fa99d47769: Download complete
2022-09-27T15:39:29.0662028Z 98a26bc0781e: Verifying Checksum
2022-09-27T15:39:29.0662392Z 98a26bc0781e: Download complete
2022-09-27T15:39:29.0924105Z 17acc9e30503: Download complete
2022-09-27T15:39:29.1914514Z e8b4222e7a59: Verifying Checksum
2022-09-27T15:39:29.1914869Z e8b4222e7a59: Download complete
2022-09-27T15:39:29.2675474Z cc8443c330a0: Verifying Checksum
2022-09-27T15:39:29.2676090Z cc8443c330a0: Download complete
2022-09-27T15:39:29.3610706Z c2fcfa2400df: Verifying Checksum
2022-09-27T15:39:29.3611084Z c2fcfa2400df: Download complete
2022-09-27T15:39:30.2083924Z dce607cbd09e: Verifying Checksum
2022-09-27T15:39:30.2084555Z dce607cbd09e: Download complete
2022-09-27T15:39:30.3212133Z 45b253446018: Verifying Checksum
2022-09-27T15:39:30.3212531Z 45b253446018: Download complete
2022-09-27T15:39:30.4309742Z 752f98c7a6d7: Verifying Checksum
2022-09-27T15:39:30.4310172Z 752f98c7a6d7: Download complete
2022-09-27T15:39:31.0777002Z 410e31c94a04: Verifying Checksum
2022-09-27T15:39:31.0777431Z 410e31c94a04: Download complete
2022-09-27T15:39:31.1696898Z aa4bb3ec24a7: Verifying Checksum
2022-09-27T15:39:31.1697322Z aa4bb3ec24a7: Download complete
2022-09-27T15:39:32.2396898Z 6a9eea4b3aa4: Download complete
2022-09-27T15:39:32.5663016Z a823f5718e87: Verifying Checksum
2022-09-27T15:39:32.5663683Z a823f5718e87: Download complete
2022-09-27T15:39:32.6474213Z 788ace045743: Download complete
2022-09-27T15:39:33.6813936Z b752992950f8: Verifying Checksum
2022-09-27T15:39:33.6814451Z b752992950f8: Download complete
2022-09-27T15:39:33.7560554Z 6cd5f9a2c4ae: Download complete
2022-09-27T15:39:33.8644551Z 577da355ab1b: Download complete
2022-09-27T15:39:33.9659386Z 649c4428b346: Verifying Checksum
2022-09-27T15:39:33.9659945Z 649c4428b346: Download complete
2022-09-27T15:39:34.0356271Z eadc05ea2cd3: Verifying Checksum
2022-09-27T15:39:34.0356905Z eadc05ea2cd3: Download complete
2022-09-27T15:39:34.4524228Z 93e5a7080833: Verifying Checksum
2022-09-27T15:39:34.4524859Z 93e5a7080833: Download complete
2022-09-27T15:39:34.5720065Z e6d72a41a09b: Verifying Checksum
2022-09-27T15:39:34.5720458Z e6d72a41a09b: Download complete
2022-09-27T15:39:36.0606800Z acb01049a64b: Verifying Checksum
2022-09-27T15:39:36.0607437Z acb01049a64b: Download complete
2022-09-27T15:39:36.1653069Z 343cc73c5973: Verifying Checksum
2022-09-27T15:39:36.1653721Z 343cc73c5973: Download complete
2022-09-27T15:39:39.6124701Z c76c6ad2ac0f: Verifying Checksum
2022-09-27T15:39:39.6125123Z c76c6ad2ac0f: Download complete
2022-09-27T15:39:39.7022187Z 5d0b32cc6f2a: Verifying Checksum
2022-09-27T15:39:39.7022608Z 5d0b32cc6f2a: Download complete
2022-09-27T15:39:39.7927616Z a6c12031bfcf: Download complete
2022-09-27T15:39:40.0541282Z 2fb24fb5f7cb: Pull complete
2022-09-27T15:39:40.1682889Z d6e4b45751c9: Pull complete
2022-09-27T15:40:02.0324913Z 98a26bc0781e: Pull complete
2022-09-27T15:40:03.9112150Z 07c42b0591b2: Pull complete
2022-09-27T15:40:05.7835507Z 9be88323b57e: Pull complete
2022-09-27T15:40:13.6586663Z 2c7b68ade49f: Pull complete
2022-09-27T15:40:15.5354607Z 44206692de1d: Pull complete
2022-09-27T15:40:17.4200160Z f751461554fa: Pull complete
2022-09-27T15:40:17.8211901Z 0c8bd29be614: Verifying Checksum
2022-09-27T15:40:17.8212301Z 0c8bd29be614: Download complete
2022-09-27T15:40:19.3254703Z 316750fef2e6: Pull complete
2022-09-27T15:40:23.6427323Z c069021d810b: Pull complete
2022-09-27T15:40:25.8009015Z e0fdd58e805b: Pull complete
2022-09-27T15:40:27.6794625Z 751286b45698: Pull complete
2022-09-27T15:40:39.2819905Z 7d69e17e7339: Verifying Checksum
2022-09-27T15:40:39.2820600Z 7d69e17e7339: Download complete
2022-09-27T15:41:05.0039610Z 0c8bd29be614: Pull complete
2022-09-27T15:41:06.8788051Z 3bb9e7ea569e: Pull complete
2022-09-27T15:41:08.7447891Z efeff9c74fbf: Pull complete
2022-09-27T15:41:10.6846426Z 81a5271d43c8: Pull complete
2022-09-27T15:41:12.5067916Z 903ca36d4d71: Pull complete
2022-09-27T15:41:14.3516887Z d52c758f8e75: Pull complete
2022-09-27T15:41:16.1955902Z a4ce2fdd9133: Pull complete
2022-09-27T15:41:20.4113180Z cae8823a1cd1: Pull complete
2022-09-27T15:41:23.2991219Z 3298fe919163: Pull complete
2022-09-27T15:41:26.7562090Z b9b9b9d06eef: Pull complete
2022-09-27T15:41:29.3781094Z 62fa99d47769: Pull complete
2022-09-27T15:41:31.8460563Z 17acc9e30503: Pull complete
2022-09-27T15:41:34.6825064Z e8b4222e7a59: Pull complete
2022-09-27T15:41:43.2348981Z b752992950f8: Pull complete
2022-09-27T15:41:45.0964638Z cc8443c330a0: Pull complete
2022-09-27T15:41:47.0057837Z c2fcfa2400df: Pull complete
2022-09-27T15:41:49.5399120Z dce607cbd09e: Pull complete
2022-09-27T15:41:51.2130797Z 45b253446018: Pull complete
2022-09-27T15:41:53.1187792Z 752f98c7a6d7: Pull complete
2022-09-27T15:41:55.4343257Z 410e31c94a04: Pull complete
2022-09-27T15:41:57.2829555Z aa4bb3ec24a7: Pull complete
2022-09-27T15:42:00.4310398Z 6a9eea4b3aa4: Pull complete
2022-09-27T15:42:00.5399550Z a823f5718e87: Pull complete
2022-09-27T15:42:00.6513793Z 788ace045743: Pull complete
2022-09-27T15:42:05.2433910Z c76c6ad2ac0f: Pull complete
2022-09-27T15:42:05.3594476Z 6cd5f9a2c4ae: Pull complete
2022-09-27T15:42:05.4649177Z 577da355ab1b: Pull complete
2022-09-27T15:42:05.5583670Z 649c4428b346: Pull complete
2022-09-27T15:42:05.6560362Z eadc05ea2cd3: Pull complete
2022-09-27T15:42:06.4679441Z 93e5a7080833: Pull complete
2022-09-27T15:42:06.5776967Z e6d72a41a09b: Pull complete
2022-09-27T15:42:08.4564265Z acb01049a64b: Pull complete
2022-09-27T15:42:08.5689790Z 343cc73c5973: Pull complete
2022-09-27T15:42:50.3652648Z 7d69e17e7339: Pull complete
2022-09-27T15:42:52.4392015Z 5d0b32cc6f2a: Pull complete
2022-09-27T15:42:54.3180731Z a6c12031bfcf: Pull complete
2022-09-27T15:42:55.6344284Z Digest: sha256:9bb261bc4d8aeb82a71b1f0709da9c979e85a12a79c4a85c3fe3adddddcb2663
2022-09-27T15:42:56.1354178Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:42:56.4175967Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:42:56.4270644Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-09-27T15:42:56.4271026Z with:
2022-09-27T15:42:56.4271402Z   timeout_minutes: 10
2022-09-27T15:42:56.4271734Z   max_attempts: 3
2022-09-27T15:42:56.4272143Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-09-27T15:42:56.4272619Z   retry_wait_seconds: 10
2022-09-27T15:42:56.4273016Z   polling_interval_seconds: 1
2022-09-27T15:42:56.4273310Z   warning_on_retry: true
2022-09-27T15:42:56.4273650Z   continue_on_error: false
2022-09-27T15:42:56.4273976Z env:
2022-09-27T15:42:56.4274232Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:42:56.4274607Z ##[endgroup]
2022-09-27T15:42:56.4831383Z 
2022-09-27T15:42:56.4896132Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-09-27T15:42:56.4898725Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.57.run ==
2022-09-27T15:42:56.4899761Z + HAS_NVIDIA_DRIVER=0
2022-09-27T15:42:56.4903325Z ++ command -v nvidia-smi
2022-09-27T15:42:56.4905472Z + '[' -x '' ']'
2022-09-27T15:42:56.4906128Z + '[' 0 -eq 0 ']'
2022-09-27T15:42:56.4906650Z + sudo yum groupinstall -y 'Development Tools'
2022-09-27T15:42:56.9670063Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:42:57.2951664Z Resolving Dependencies
2022-09-27T15:42:57.2956772Z --> Running transaction check
2022-09-27T15:42:57.2959665Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-09-27T15:42:57.3182506Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-09-27T15:42:57.5312074Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-09-27T15:42:57.5313350Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-09-27T15:42:57.5361337Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-09-27T15:42:57.5368503Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-09-27T15:42:57.5379842Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-09-27T15:42:57.5494866Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-09-27T15:42:57.5502149Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-09-27T15:42:57.5547268Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-09-27T15:42:57.5572580Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-09-27T15:42:57.5581556Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-09-27T15:42:57.5589371Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-09-27T15:42:57.5685751Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-09-27T15:42:57.5829541Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-09-27T15:42:57.5848870Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.6024451Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6044586Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6101447Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6154814Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6212402Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6266756Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6322531Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6378650Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6530282Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6551757Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6573799Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.6601221Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.6635802Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-09-27T15:42:57.6701318Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-09-27T15:42:57.6716125Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-09-27T15:42:57.6769934Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-09-27T15:42:57.6784728Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-09-27T15:42:57.6804086Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-09-27T15:42:57.6835740Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-09-27T15:42:57.6872499Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-09-27T15:42:57.6898602Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-09-27T15:42:57.6932706Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-09-27T15:42:57.7178825Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-09-27T15:42:57.7197956Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-09-27T15:42:57.7221876Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-09-27T15:42:57.7395653Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7415561Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7416886Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7417602Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7418343Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7419247Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7419904Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7420737Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7421547Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7422182Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7422830Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7423463Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7424199Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7424934Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7425559Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7426357Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7445097Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7466825Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:42:57.7491955Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-09-27T15:42:57.7514451Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-09-27T15:42:57.7561731Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:42:57.7580433Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:42:57.7593106Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:42:57.7760966Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:42:57.7774774Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:57.7789980Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:57.7804153Z --> Running transaction check
2022-09-27T15:42:57.7807498Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-09-27T15:42:57.7882335Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-09-27T15:42:57.7920981Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-09-27T15:42:57.7936035Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8013195Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-09-27T15:42:57.8023744Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-09-27T15:42:57.8024925Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-09-27T15:42:57.8098467Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-09-27T15:42:57.8164575Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-09-27T15:42:57.8174305Z ---> Package glibc-devel.x86_64 0:2.26-60.amzn2 will be installed
2022-09-27T15:42:57.8293833Z --> Processing Dependency: glibc-headers = 2.26-60.amzn2 for package: glibc-devel-2.26-60.amzn2.x86_64
2022-09-27T15:42:57.8320643Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-60.amzn2.x86_64
2022-09-27T15:42:57.8321445Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1 will be installed
2022-09-27T15:42:57.8326567Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8340485Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8367839Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8403864Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8420365Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-09-27T15:42:57.8432878Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8447599Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8473444Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:42:57.8519585Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-09-27T15:42:57.8535242Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-09-27T15:42:57.8556471Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-09-27T15:42:57.8633551Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:42:57.8672051Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:42:57.8673133Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:42:57.8693235Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:42:57.8711395Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:42:57.8717727Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-09-27T15:42:57.8767577Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-09-27T15:42:57.8865131Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-09-27T15:42:57.8877719Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-09-27T15:42:57.8893409Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-09-27T15:42:57.8894597Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-09-27T15:42:57.8922982Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:42:57.9130295Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:57.9144712Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:57.9171169Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:57.9171937Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:42:57.9297632Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:42:58.0311402Z --> Running transaction check
2022-09-27T15:42:58.0312221Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-09-27T15:42:58.0322237Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-09-27T15:42:58.0348633Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-09-27T15:42:58.0349751Z ---> Package glibc-headers.x86_64 0:2.26-60.amzn2 will be installed
2022-09-27T15:42:58.0424322Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-60.amzn2.x86_64
2022-09-27T15:42:58.1515360Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-60.amzn2.x86_64
2022-09-27T15:42:58.1516226Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-09-27T15:42:58.1584308Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-09-27T15:42:58.1611273Z ---> Package kernel-devel.x86_64 0:4.14.291-218.527.amzn2 will be installed
2022-09-27T15:42:58.1639103Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.291-218.527.amzn2.x86_64
2022-09-27T15:42:58.1660039Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-09-27T15:42:58.1688368Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-09-27T15:42:58.1707370Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-09-27T15:42:58.1757243Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:42:58.1778668Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:42:58.1779730Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:42:58.1780574Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-09-27T15:42:58.1794005Z --> Running transaction check
2022-09-27T15:42:58.1795097Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-09-27T15:42:58.1813136Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-09-27T15:42:58.1825496Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-09-27T15:42:58.1849455Z ---> Package kernel-headers.x86_64 0:4.14.291-218.527.amzn2 will be installed
2022-09-27T15:42:58.1850918Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-09-27T15:42:58.1868739Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-09-27T15:42:58.1927416Z --> Running transaction check
2022-09-27T15:42:58.1927885Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1 will be installed
2022-09-27T15:42:58.4592036Z --> Finished Dependency Resolution
2022-09-27T15:42:58.5379786Z 
2022-09-27T15:42:58.5379959Z Dependencies Resolved
2022-09-27T15:42:58.5499682Z 
2022-09-27T15:42:58.5500079Z ================================================================================
2022-09-27T15:42:58.5500474Z  Package                 Arch     Version                    Repository    Size
2022-09-27T15:42:58.5500844Z ================================================================================
2022-09-27T15:42:58.5502592Z Installing for group install "Development Tools":
2022-09-27T15:42:58.5503222Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-09-27T15:42:58.5503706Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-09-27T15:42:58.5504148Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-09-27T15:42:58.5504590Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-09-27T15:42:58.5505035Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-09-27T15:42:58.5505482Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-09-27T15:42:58.5505935Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-09-27T15:42:58.5506380Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-09-27T15:42:58.5506835Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-09-27T15:42:58.5507247Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-09-27T15:42:58.5507686Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-09-27T15:42:58.5508102Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-09-27T15:42:58.5508580Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-09-27T15:42:58.5509836Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-09-27T15:42:58.5510280Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-09-27T15:42:58.5510709Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-09-27T15:42:58.5511201Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-09-27T15:42:58.5511583Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-09-27T15:42:58.5512097Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-09-27T15:42:58.5512521Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-09-27T15:42:58.5512957Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-09-27T15:42:58.5513368Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-09-27T15:42:58.5513804Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-09-27T15:42:58.5514277Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-09-27T15:42:58.5514723Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-09-27T15:42:58.5515072Z Installing for dependencies:
2022-09-27T15:42:58.5515483Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-09-27T15:42:58.5515895Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-09-27T15:42:58.5516325Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-09-27T15:42:58.5516949Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-09-27T15:42:58.5517390Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-09-27T15:42:58.5517886Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-09-27T15:42:58.5518335Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-09-27T15:42:58.5518803Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-09-27T15:42:58.5519258Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-09-27T15:42:58.5519718Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-09-27T15:42:58.5520173Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-09-27T15:42:58.5520646Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-09-27T15:42:58.5521081Z  glibc-devel             x86_64   2.26-60.amzn2              amzn2-core   994 k
2022-09-27T15:42:58.5521533Z  glibc-headers           x86_64   2.26-60.amzn2              amzn2-core   515 k
2022-09-27T15:42:58.5521977Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-09-27T15:42:58.5522431Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.1        amzn2-core    23 k
2022-09-27T15:42:58.5522899Z  kernel-devel            x86_64   4.14.291-218.527.amzn2     amzn2-core    13 M
2022-09-27T15:42:58.5523336Z  kernel-headers          x86_64   4.14.291-218.527.amzn2     amzn2-core   1.2 M
2022-09-27T15:42:58.5523783Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-09-27T15:42:58.5524430Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-09-27T15:42:58.5524892Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-09-27T15:42:58.5525322Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-09-27T15:42:58.5525746Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-09-27T15:42:58.5526184Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-09-27T15:42:58.5526616Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-09-27T15:42:58.5527057Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-09-27T15:42:58.5527477Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-09-27T15:42:58.5527921Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-09-27T15:42:58.5528357Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-09-27T15:42:58.5528815Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-09-27T15:42:58.5529225Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-09-27T15:42:58.5529960Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-09-27T15:42:58.5530418Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-09-27T15:42:58.5530865Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-09-27T15:42:58.5531339Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-09-27T15:42:58.5531812Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-09-27T15:42:58.5532296Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-09-27T15:42:58.5532753Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-09-27T15:42:58.5533232Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-09-27T15:42:58.5533697Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-09-27T15:42:58.5534141Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-09-27T15:42:58.5534704Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-09-27T15:42:58.5535143Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.1         amzn2-core    50 k
2022-09-27T15:42:58.5535420Z 
2022-09-27T15:42:58.5535550Z Transaction Summary
2022-09-27T15:42:58.5535827Z ================================================================================
2022-09-27T15:42:58.5536157Z Install  25 Packages (+43 Dependent packages)
2022-09-27T15:42:58.5536362Z 
2022-09-27T15:42:58.5536486Z Total download size: 96 M
2022-09-27T15:42:58.5536739Z Installed size: 303 M
2022-09-27T15:42:58.5537014Z Downloading packages:
2022-09-27T15:42:58.5555474Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-09-27T15:43:00.0380587Z --------------------------------------------------------------------------------
2022-09-27T15:43:00.0381058Z Total                                               65 MB/s |  96 MB  00:01     
2022-09-27T15:43:00.1470275Z Running transaction check
2022-09-27T15:43:00.2251330Z Running transaction test
2022-09-27T15:43:02.6220111Z Transaction test succeeded
2022-09-27T15:43:02.6223307Z Running transaction
2022-09-27T15:43:07.9363231Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-09-27T15:43:10.4286341Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-09-27T15:43:12.8939604Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-09-27T15:43:15.3232466Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-09-27T15:43:17.8086592Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-09-27T15:43:18.1098156Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-09-27T15:43:18.1541563Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-09-27T15:43:18.1951385Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-09-27T15:43:18.2202016Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-09-27T15:43:18.3045991Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-09-27T15:43:19.3699276Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-09-27T15:43:19.4079894Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-09-27T15:43:19.4298375Z   Installing : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       13/68 
2022-09-27T15:43:19.4523636Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 14/68 
2022-09-27T15:43:19.4880773Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         15/68 
2022-09-27T15:43:19.5498279Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-09-27T15:43:19.6115836Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   17/68 
2022-09-27T15:43:19.7237284Z   Installing : elfutils-0.176-2.amzn2.x86_64                              18/68 
2022-09-27T15:43:19.7565334Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         19/68 
2022-09-27T15:43:19.7820482Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 20/68 
2022-09-27T15:43:19.8175248Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          21/68 
2022-09-27T15:43:19.8519583Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        22/68 
2022-09-27T15:43:19.9370261Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-09-27T15:43:20.1174439Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-09-27T15:43:20.4237527Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-09-27T15:43:20.5951314Z   Installing : kernel-headers-4.14.291-218.527.amzn2.x86_64               26/68 
2022-09-27T15:43:20.7294303Z   Installing : glibc-headers-2.26-60.amzn2.x86_64                         27/68 
2022-09-27T15:43:20.7719124Z   Installing : glibc-devel-2.26-60.amzn2.x86_64                           28/68 
2022-09-27T15:43:21.1700290Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               29/68 
2022-09-27T15:43:21.2033191Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              30/68 
2022-09-27T15:43:21.2390153Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               31/68 
2022-09-27T15:43:21.2696141Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      32/68 
2022-09-27T15:43:21.2968623Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               33/68 
2022-09-27T15:43:21.3220971Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  34/68 
2022-09-27T15:43:21.4168425Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  35/68 
2022-09-27T15:43:21.4751730Z   Installing : autoconf-2.69-11.amzn2.noarch                              36/68 
2022-09-27T15:43:21.5543386Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               37/68 
2022-09-27T15:43:21.6426014Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      38/68 
2022-09-27T15:43:21.7521900Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      39/68 
2022-09-27T15:43:21.7879624Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           40/68 
2022-09-27T15:43:21.8200031Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            41/68 
2022-09-27T15:43:24.0060120Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-09-27T15:43:31.1844894Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-09-27T15:43:42.4780771Z   Installing : kernel-devel-4.14.291-218.527.amzn2.x86_64                 44/68 
2022-09-27T15:43:43.1102438Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-09-27T15:43:43.1666718Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-09-27T15:43:43.2256137Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-09-27T15:43:43.3614040Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-09-27T15:43:43.5409172Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-09-27T15:43:43.6406513Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-09-27T15:43:44.8620424Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-09-27T15:43:46.4965533Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-09-27T15:43:46.6145334Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-09-27T15:43:46.6577337Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-09-27T15:43:46.7000745Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-09-27T15:43:46.7610729Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-09-27T15:43:46.8261857Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-09-27T15:43:46.9365070Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-09-27T15:43:47.0064297Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-09-27T15:43:47.0590228Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-09-27T15:43:47.1028654Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-09-27T15:43:47.1536479Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-09-27T15:43:47.8674576Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-09-27T15:43:47.9172597Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-09-27T15:43:47.9514071Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-09-27T15:43:48.2836281Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-09-27T15:43:48.3362929Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-09-27T15:43:48.4082031Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-09-27T15:43:48.4223079Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-09-27T15:43:48.4358399Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-09-27T15:43:48.4475469Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-09-27T15:43:48.4599014Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-09-27T15:43:48.4727090Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              5/68 
2022-09-27T15:43:48.4854200Z   Verifying  : glibc-headers-2.26-60.amzn2.x86_64                          6/68 
2022-09-27T15:43:48.4976317Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-09-27T15:43:48.5090375Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-09-27T15:43:48.5205439Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                          9/68 
2022-09-27T15:43:48.5329094Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            10/68 
2022-09-27T15:43:48.5449713Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      11/68 
2022-09-27T15:43:48.5603926Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            12/68 
2022-09-27T15:43:48.5701457Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           13/68 
2022-09-27T15:43:48.5820300Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  14/68 
2022-09-27T15:43:48.5925147Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        15/68 
2022-09-27T15:43:48.6066849Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            16/68 
2022-09-27T15:43:48.6184240Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           17/68 
2022-09-27T15:43:48.6300758Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             18/68 
2022-09-27T15:43:48.6409598Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              19/68 
2022-09-27T15:43:48.6560331Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            20/68 
2022-09-27T15:43:48.6694430Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               21/68 
2022-09-27T15:43:48.6829494Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     22/68 
2022-09-27T15:43:48.6966344Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      23/68 
2022-09-27T15:43:48.7095989Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              24/68 
2022-09-27T15:43:48.7212980Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          25/68 
2022-09-27T15:43:48.7344842Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             26/68 
2022-09-27T15:43:48.7446138Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          27/68 
2022-09-27T15:43:48.7602082Z   Verifying  : glibc-devel-2.26-60.amzn2.x86_64                           28/68 
2022-09-27T15:43:48.7779952Z   Verifying  : kernel-devel-4.14.291-218.527.amzn2.x86_64                 29/68 
2022-09-27T15:43:48.7892424Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  30/68 
2022-09-27T15:43:48.8020095Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       31/68 
2022-09-27T15:43:48.8147090Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               32/68 
2022-09-27T15:43:48.8238539Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/68 
2022-09-27T15:43:48.8351666Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      34/68 
2022-09-27T15:43:48.8459725Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           35/68 
2022-09-27T15:43:48.8578155Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 36/68 
2022-09-27T15:43:48.8689982Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           37/68 
2022-09-27T15:43:48.8815921Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      38/68 
2022-09-27T15:43:48.8915761Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               39/68 
2022-09-27T15:43:48.9038804Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          40/68 
2022-09-27T15:43:48.9164445Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             41/68 
2022-09-27T15:43:48.9285648Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          42/68 
2022-09-27T15:43:48.9395443Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              43/68 
2022-09-27T15:43:48.9520989Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        44/68 
2022-09-27T15:43:48.9639461Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               45/68 
2022-09-27T15:43:48.9761577Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        46/68 
2022-09-27T15:43:48.9887841Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  47/68 
2022-09-27T15:43:48.9999905Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          48/68 
2022-09-27T15:43:49.0138646Z   Verifying  : kernel-headers-4.14.291-218.527.amzn2.x86_64               49/68 
2022-09-27T15:43:49.0293320Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-09-27T15:43:49.0422176Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-09-27T15:43:49.0541012Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-09-27T15:43:49.0645000Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-09-27T15:43:49.0753140Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-09-27T15:43:49.0878602Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-09-27T15:43:49.1002404Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-09-27T15:43:49.1123885Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-09-27T15:43:49.1250377Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-09-27T15:43:49.1379322Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    59/68 
2022-09-27T15:43:49.1487751Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          60/68 
2022-09-27T15:43:49.1598743Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               61/68 
2022-09-27T15:43:49.1737075Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 62/68 
2022-09-27T15:43:49.1864997Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         63/68 
2022-09-27T15:43:49.1977324Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              64/68 
2022-09-27T15:43:49.2090362Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              65/68 
2022-09-27T15:43:49.2217693Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   66/68 
2022-09-27T15:43:49.2342273Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         67/68 
2022-09-27T15:43:49.3160439Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       68/68 
2022-09-27T15:43:49.3162356Z 
2022-09-27T15:43:49.3162786Z Installed:
2022-09-27T15:43:49.3164718Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-09-27T15:43:49.3165889Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-09-27T15:43:49.3166347Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-09-27T15:43:49.3166930Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-09-27T15:43:49.3170767Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-09-27T15:43:49.3171314Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-09-27T15:43:49.3171765Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-09-27T15:43:49.3172121Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-09-27T15:43:49.3172556Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-09-27T15:43:49.3172989Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-09-27T15:43:49.3173420Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-09-27T15:43:49.3174559Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-09-27T15:43:49.3175017Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-09-27T15:43:49.3175468Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-09-27T15:43:49.3175875Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-09-27T15:43:49.3176260Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-09-27T15:43:49.3176711Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-09-27T15:43:49.3177151Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-09-27T15:43:49.3177528Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-09-27T15:43:49.3178020Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-09-27T15:43:49.3178461Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-09-27T15:43:49.3178902Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-09-27T15:43:49.3179316Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-09-27T15:43:49.3179764Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-09-27T15:43:49.3180164Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-09-27T15:43:49.3180374Z 
2022-09-27T15:43:49.3180531Z Dependency Installed:
2022-09-27T15:43:49.3180913Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-09-27T15:43:49.3181342Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-09-27T15:43:49.3181785Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-09-27T15:43:49.3182204Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-09-27T15:43:49.3182637Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-09-27T15:43:49.3183050Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-09-27T15:43:49.3183482Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-09-27T15:43:49.3183918Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-09-27T15:43:49.3184390Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-09-27T15:43:49.3184830Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-09-27T15:43:49.3185256Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-09-27T15:43:49.3185727Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-09-27T15:43:49.3186361Z   glibc-devel.x86_64 0:2.26-60.amzn2                                            
2022-09-27T15:43:49.3186803Z   glibc-headers.x86_64 0:2.26-60.amzn2                                          
2022-09-27T15:43:49.3187214Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-09-27T15:43:49.3187741Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1                                   
2022-09-27T15:43:49.3188213Z   kernel-devel.x86_64 0:4.14.291-218.527.amzn2                                  
2022-09-27T15:43:49.3188632Z   kernel-headers.x86_64 0:4.14.291-218.527.amzn2                                
2022-09-27T15:43:49.3189077Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-09-27T15:43:49.3189504Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-09-27T15:43:49.3189925Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-09-27T15:43:49.3190325Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-09-27T15:43:49.3190750Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-09-27T15:43:49.3191171Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-09-27T15:43:49.3191567Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-09-27T15:43:49.3191986Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-09-27T15:43:49.3192422Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-09-27T15:43:49.3192843Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-09-27T15:43:49.3193240Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-09-27T15:43:49.3193649Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-09-27T15:43:49.3194079Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-09-27T15:43:49.3194482Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-09-27T15:43:49.3194904Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-09-27T15:43:49.3195354Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-09-27T15:43:49.3195827Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-09-27T15:43:49.3196278Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-09-27T15:43:49.3196754Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-09-27T15:43:49.3197223Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-09-27T15:43:49.3197664Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-09-27T15:43:49.3198115Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-09-27T15:43:49.3198572Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-09-27T15:43:49.3199015Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-09-27T15:43:49.3199421Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1                                        
2022-09-27T15:43:49.3199628Z 
2022-09-27T15:43:49.3199737Z Complete!
2022-09-27T15:43:49.3595148Z ++ uname -r
2022-09-27T15:43:49.3601003Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-09-27T15:43:49.8652598Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:43:50.1431866Z Resolving Dependencies
2022-09-27T15:43:50.1437960Z --> Running transaction check
2022-09-27T15:43:50.1438444Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-09-27T15:43:50.4297080Z --> Finished Dependency Resolution
2022-09-27T15:43:50.5109728Z 
2022-09-27T15:43:50.5110302Z Dependencies Resolved
2022-09-27T15:43:50.5115586Z 
2022-09-27T15:43:50.5116002Z ================================================================================
2022-09-27T15:43:50.5116423Z  Package          Arch       Version                       Repository      Size
2022-09-27T15:43:50.5116865Z ================================================================================
2022-09-27T15:43:50.5117161Z Installing:
2022-09-27T15:43:50.5117717Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-09-27T15:43:50.5117954Z 
2022-09-27T15:43:50.5118064Z Transaction Summary
2022-09-27T15:43:50.5118354Z ================================================================================
2022-09-27T15:43:50.5118667Z Install  1 Package
2022-09-27T15:43:50.5118815Z 
2022-09-27T15:43:50.5118953Z Total download size: 13 M
2022-09-27T15:43:50.5119209Z Installed size: 60 M
2022-09-27T15:43:50.5119489Z Downloading packages:
2022-09-27T15:43:50.5128446Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-09-27T15:43:50.9594187Z Running transaction check
2022-09-27T15:43:50.9779010Z Running transaction test
2022-09-27T15:43:51.3834348Z Transaction test succeeded
2022-09-27T15:43:51.3837286Z Running transaction
2022-09-27T15:44:06.8617221Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-09-27T15:44:06.9478617Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-09-27T15:44:06.9478924Z 
2022-09-27T15:44:06.9479053Z Installed:
2022-09-27T15:44:06.9479470Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-09-27T15:44:06.9479692Z 
2022-09-27T15:44:06.9479816Z Complete!
2022-09-27T15:44:06.9832500Z + sudo modprobe backlight
2022-09-27T15:44:07.0052596Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.57.run
2022-09-27T15:44:14.0328269Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-09-27T15:44:15.4459268Z Verifying archive integrity... OK
2022-09-27T15:44:42.1015772Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.57................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-09-27T15:44:42.2458062Z 
2022-09-27T15:44:42.2461054Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-09-27T15:44:42.2461434Z 
2022-09-27T15:44:58.2149693Z 
2022-09-27T15:44:58.2151292Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-09-27T15:44:58.2151937Z 
2022-09-27T15:45:07.1044041Z + sudo rm -fv /tmp/nvidia_driver
2022-09-27T15:45:07.2034128Z removed ‘/tmp/nvidia_driver’
2022-09-27T15:45:07.2047774Z + nvidia-smi
2022-09-27T15:45:12.0697616Z Tue Sep 27 15:45:12 2022       
2022-09-27T15:45:12.0698251Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:12.0699119Z | NVIDIA-SMI 515.57       Driver Version: 515.57       CUDA Version: 11.7     |
2022-09-27T15:45:12.0699658Z |-------------------------------+----------------------+----------------------+
2022-09-27T15:45:12.0700981Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-09-27T15:45:12.0701562Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-09-27T15:45:12.0701967Z |                               |                      |               MIG M. |
2022-09-27T15:45:12.0702294Z |===============================+======================+======================|
2022-09-27T15:45:12.0757028Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           2669329593 |
2022-09-27T15:45:12.0757451Z | N/A   30C    P0    39W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-09-27T15:45:12.0757789Z |                               |                      |                  N/A |
2022-09-27T15:45:12.0758269Z +-------------------------------+----------------------+----------------------+
2022-09-27T15:45:12.0803801Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |           6196437890 |
2022-09-27T15:45:12.0804335Z | N/A   35C    P0    37W / 150W |      0MiB /  7680MiB |     97%      Default |
2022-09-27T15:45:12.0804674Z |                               |                      |                  N/A |
2022-09-27T15:45:12.0805142Z +-------------------------------+----------------------+----------------------+
2022-09-27T15:45:12.0805494Z                                                                                
2022-09-27T15:45:12.0805938Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:12.0806329Z | Processes:                                                                  |
2022-09-27T15:45:12.0806669Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-09-27T15:45:12.0807018Z |        ID   ID                                                   Usage      |
2022-09-27T15:45:12.0807339Z |=============================================================================|
2022-09-27T15:45:12.0808528Z |  No running processes found                                                 |
2022-09-27T15:45:12.0808984Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:12.6193638Z == Installing nvidia container toolkit for amzn2 ==
2022-09-27T15:45:12.6197649Z + sudo yum install -y yum-utils
2022-09-27T15:45:13.1673747Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:13.4344046Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-09-27T15:45:13.4344463Z Nothing to do
2022-09-27T15:45:13.4566134Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-09-27T15:45:13.9970623Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:14.0319126Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-09-27T15:45:14.0319843Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-09-27T15:45:14.0320364Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-09-27T15:45:14.0482964Z + sudo yum install -y nvidia-docker2
2022-09-27T15:45:14.5914242Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:14.6515608Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-09-27T15:45:14.6614490Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:14.6614896Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:14.6615312Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:14.6615781Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-09-27T15:45:15.0578745Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-09-27T15:45:15.0691870Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:15.0692275Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:15.0692663Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:15.0693324Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-09-27T15:45:15.2944703Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-09-27T15:45:15.3029644Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:15.3030263Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:15.3030676Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:15.3031154Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-09-27T15:45:17.0925372Z Resolving Dependencies
2022-09-27T15:45:17.0931941Z --> Running transaction check
2022-09-27T15:45:17.0932820Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-09-27T15:45:17.0958702Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-09-27T15:45:17.1365565Z --> Running transaction check
2022-09-27T15:45:17.1366070Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:17.1514916Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:17.1525445Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:17.1655804Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:17.1656447Z --> Running transaction check
2022-09-27T15:45:17.1657300Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:17.1667682Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:17.1695313Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:17.1696060Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:17.1696842Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:17.1699942Z --> Running transaction check
2022-09-27T15:45:17.1700585Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:17.4875521Z --> Finished Dependency Resolution
2022-09-27T15:45:17.5634262Z 
2022-09-27T15:45:17.5634715Z Dependencies Resolved
2022-09-27T15:45:17.5648444Z 
2022-09-27T15:45:17.5648721Z ================================================================================
2022-09-27T15:45:17.5649080Z  Package                        Arch    Version      Repository            Size
2022-09-27T15:45:17.5649466Z ================================================================================
2022-09-27T15:45:17.5649962Z Installing:
2022-09-27T15:45:17.5650504Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-09-27T15:45:17.5650845Z Installing for dependencies:
2022-09-27T15:45:17.5651332Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-09-27T15:45:17.5651870Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-09-27T15:45:17.5653148Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-09-27T15:45:17.5653758Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-09-27T15:45:17.5653982Z 
2022-09-27T15:45:17.5654108Z Transaction Summary
2022-09-27T15:45:17.5654427Z ================================================================================
2022-09-27T15:45:17.5654783Z Install  1 Package (+4 Dependent packages)
2022-09-27T15:45:17.5655794Z 
2022-09-27T15:45:17.5656175Z Total download size: 4.3 M
2022-09-27T15:45:17.5656460Z Installed size: 12 M
2022-09-27T15:45:17.5656730Z Downloading packages:
2022-09-27T15:45:17.6830230Z --------------------------------------------------------------------------------
2022-09-27T15:45:17.6830931Z Total                                               37 MB/s | 4.3 MB  00:00     
2022-09-27T15:45:17.6878769Z Running transaction check
2022-09-27T15:45:17.7052406Z Running transaction test
2022-09-27T15:45:17.7215384Z Transaction test succeeded
2022-09-27T15:45:17.7218633Z Running transaction
2022-09-27T15:45:18.2223700Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-09-27T15:45:18.2636404Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-09-27T15:45:18.3769795Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-09-27T15:45:18.4018917Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-09-27T15:45:18.4405775Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-09-27T15:45:18.4527975Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-09-27T15:45:18.4636440Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-09-27T15:45:18.4725540Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-09-27T15:45:18.4833378Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-09-27T15:45:18.5591105Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-09-27T15:45:18.5591522Z 
2022-09-27T15:45:18.5591721Z Installed:
2022-09-27T15:45:18.5592245Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-09-27T15:45:18.5592463Z 
2022-09-27T15:45:18.5592560Z Dependency Installed:
2022-09-27T15:45:18.5593003Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-09-27T15:45:18.5593493Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-09-27T15:45:18.5593959Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-09-27T15:45:18.5594475Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-09-27T15:45:18.5594754Z 
2022-09-27T15:45:18.5594860Z Complete!
2022-09-27T15:45:18.6850914Z + sudo systemctl restart docker
2022-09-27T15:45:26.4870081Z + echo 'GPU_FLAG=--gpus all'
2022-09-27T15:45:26.6313295Z Command completed after 1 attempt(s).
2022-09-27T15:45:26.6313524Z 
2022-09-27T15:45:26.6373307Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-09-27T15:45:26.6373738Z python3 -m pip install psutil==5.9.1
2022-09-27T15:45:26.6374094Z python3 -m pip install pynvml==11.4.1
2022-09-27T15:45:26.6374459Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-09-27T15:45:26.6374876Z echo "::set-output name=monitor-script-pid::${!}"
2022-09-27T15:45:26.6388554Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:45:26.6388840Z env:
2022-09-27T15:45:26.6389102Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:26.6389382Z   GPU_FLAG: --gpus all
2022-09-27T15:45:26.6389620Z ##[endgroup]
2022-09-27T15:45:27.4472947Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:45:27.8173067Z Collecting psutil==5.9.1
2022-09-27T15:45:27.8384518Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-09-27T15:45:27.9093413Z Installing collected packages: psutil
2022-09-27T15:45:28.0682742Z Successfully installed psutil-5.9.1
2022-09-27T15:45:28.5417596Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:45:28.6295075Z Collecting pynvml==11.4.1
2022-09-27T15:45:28.6459632Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-09-27T15:45:28.6960899Z Installing collected packages: pynvml
2022-09-27T15:45:28.7499618Z Successfully installed pynvml-11.4.1
2022-09-27T15:45:28.8047651Z Prepare all required actions
2022-09-27T15:45:28.8048039Z Getting action download info
2022-09-27T15:45:28.9556816Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5)
2022-09-27T15:45:29.1638737Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-09-27T15:45:29.2981488Z ##[group]Run ./.github/actions/download-build-artifacts
2022-09-27T15:45:29.2981808Z with:
2022-09-27T15:45:29.2982112Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:45:29.2982406Z env:
2022-09-27T15:45:29.2982640Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:29.2982918Z   GPU_FLAG: --gpus all
2022-09-27T15:45:29.2983184Z ##[endgroup]
2022-09-27T15:45:29.3041788Z ##[group]Run seemethere/download-artifact-s3@v4
2022-09-27T15:45:29.3042338Z with:
2022-09-27T15:45:29.3042830Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:45:29.3043429Z   s3-bucket: gha-artifacts
2022-09-27T15:45:29.3044060Z   region: us-east-1
2022-09-27T15:45:29.3044474Z env:
2022-09-27T15:45:29.3044926Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:29.3045396Z   GPU_FLAG: --gpus all
2022-09-27T15:45:29.3045810Z ##[endgroup]
2022-09-27T15:45:30.3604322Z Found 1 objects with prefix pytorch/pytorch/3133193930/linux-bionic-cuda11.6-py3.10-gcc7/
2022-09-27T15:45:30.3604887Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:45:39.6324014Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:45:39.6324372Z 
2022-09-27T15:45:39.6325475Z Artifact download has finished successfully
2022-09-27T15:45:39.6477147Z ##[group]Run unzip -o artifacts.zip
2022-09-27T15:45:39.6477456Z unzip -o artifacts.zip
2022-09-27T15:45:39.6491612Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:45:39.6491936Z env:
2022-09-27T15:45:39.6492182Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:39.6492485Z   GPU_FLAG: --gpus all
2022-09-27T15:45:39.6492749Z ##[endgroup]
2022-09-27T15:45:39.6567550Z Archive:  artifacts.zip
2022-09-27T15:45:39.6569480Z    creating: dist/
2022-09-27T15:45:41.7190600Z   inflating: dist/torch-1.13.0a0+git52424e2-cp310-cp310-linux_x86_64.whl  
2022-09-27T15:45:41.7191033Z    creating: build/custom_test_artifacts/
2022-09-27T15:45:41.7191486Z    creating: build/custom_test_artifacts/custom-op-build/
2022-09-27T15:45:41.7191974Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-09-27T15:45:41.7198697Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:41.7199230Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-09-27T15:45:41.7199801Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:41.7200375Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:41.7200982Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:41.7203398Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:41.7204545Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:41.7205124Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:41.7205697Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:41.7208463Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:41.7209862Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:41.7211835Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:41.7212867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:41.7213932Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:41.7215058Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:41.7215651Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:41.7216242Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:41.7271197Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:41.7271906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:41.7272657Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:41.7273405Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:41.7274401Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:41.7275096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:41.7275813Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:41.7276523Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:41.7277642Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:41.7320104Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:41.7361972Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:41.7363052Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:41.7363881Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:41.7364529Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:41.7365283Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:41.7366272Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:41.7367265Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:41.7369261Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:41.7442937Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:41.7516561Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:41.7517187Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:41.7517756Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:41.7518464Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:41.7519028Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:41.7519796Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-09-27T15:45:41.7520409Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-09-27T15:45:41.7521034Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-09-27T15:45:41.7521645Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-09-27T15:45:41.7522205Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-09-27T15:45:41.7522810Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:45:41.7523857Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-09-27T15:45:41.7524478Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-09-27T15:45:41.7525252Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-09-27T15:45:41.7525878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-09-27T15:45:41.7547324Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-09-27T15:45:41.7662021Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-09-27T15:45:41.7662582Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-09-27T15:45:41.7663190Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-09-27T15:45:41.7663842Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-09-27T15:45:41.7664474Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-09-27T15:45:41.7665069Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-09-27T15:45:41.7665855Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:45:41.7666796Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-09-27T15:45:41.7667613Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-09-27T15:45:41.7668226Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-09-27T15:45:41.7668850Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-09-27T15:45:41.7690004Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-09-27T15:45:41.7771988Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-09-27T15:45:41.7772633Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:41.7773255Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:41.7773829Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-09-27T15:45:41.7774600Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-09-27T15:45:41.7775850Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:41.7776512Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-09-27T15:45:41.7779399Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-09-27T15:45:41.7780167Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-09-27T15:45:41.7781027Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-09-27T15:45:41.7874135Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-09-27T15:45:41.7936366Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-09-27T15:45:41.7936857Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-09-27T15:45:41.7937330Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-09-27T15:45:41.7944046Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:41.7944572Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-09-27T15:45:41.7945129Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:41.7945690Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:41.7946259Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:41.7948413Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:41.7949564Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:41.7950142Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:41.7950709Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:41.7953301Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:41.7954476Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:41.7956456Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:41.7957087Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:41.7958558Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:41.7959677Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:41.7960273Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:41.7960841Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:41.8016251Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:41.8016991Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:41.8017727Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:41.8018477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:41.8019219Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:41.8019899Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:41.8020602Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:41.8021295Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:41.8022166Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:41.8064743Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:41.8106633Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:41.8107618Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:41.8109055Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:41.8109689Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:41.8110362Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:41.8111465Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:41.8112424Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:41.8114441Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:41.8188223Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:41.8261764Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:41.8262383Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:41.8262940Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:41.8263731Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:41.8264551Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:41.8265157Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-09-27T15:45:41.8265772Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-09-27T15:45:41.8266405Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-09-27T15:45:41.8267586Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-09-27T15:45:41.8268579Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-09-27T15:45:41.8269240Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-09-27T15:45:41.8270059Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-09-27T15:45:41.8270739Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-09-27T15:45:41.8271420Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-09-27T15:45:41.8272075Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-09-27T15:45:41.8293157Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-09-27T15:45:41.8356764Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-09-27T15:45:41.8357501Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:41.8358197Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:41.8358887Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-09-27T15:45:41.8359517Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-09-27T15:45:41.8360570Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:41.8361190Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-09-27T15:45:41.8364230Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-09-27T15:45:41.8364839Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-09-27T15:45:41.8365613Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-09-27T15:45:41.8415480Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-09-27T15:45:41.8416072Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-09-27T15:45:41.8416737Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-09-27T15:45:41.8423418Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:41.8424030Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-09-27T15:45:41.8424700Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:41.8425392Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:41.8426125Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:41.8427386Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:41.8428948Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:41.8429626Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:41.8430356Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:41.8432725Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:41.8433874Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:41.8435443Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:41.8436467Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:41.8437829Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:41.8438964Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:41.8439656Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:41.8440341Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:41.8495512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:41.8496295Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:41.8497142Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:41.8497988Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:41.8498902Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:41.8499650Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:41.8500454Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:41.8501265Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:41.8503067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:41.8570307Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:41.8611588Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:41.8612674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:41.8613492Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:41.8656084Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:41.8656940Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:41.8657644Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:41.8658320Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:41.8658987Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:41.8692430Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:41.8766209Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:41.8766895Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:41.8767421Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:41.8769273Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:41.8770029Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:41.8770658Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-09-27T15:45:41.8771265Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-09-27T15:45:41.8771936Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-09-27T15:45:41.8772539Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-09-27T15:45:41.8773116Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-09-27T15:45:41.8773729Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:45:41.8774876Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-09-27T15:45:41.8775681Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-09-27T15:45:41.8776359Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-09-27T15:45:41.8776985Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-09-27T15:45:41.8781735Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-09-27T15:45:41.8929836Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-09-27T15:45:41.8930634Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-09-27T15:45:41.8931512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-09-27T15:45:41.8932209Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-09-27T15:45:41.8932881Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-09-27T15:45:41.8933519Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-09-27T15:45:41.8934148Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:45:41.8935058Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-09-27T15:45:41.8935706Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-09-27T15:45:41.8936381Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-09-27T15:45:41.8937010Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-09-27T15:45:41.8957788Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-09-27T15:45:41.9015934Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-09-27T15:45:41.9016669Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:41.9017318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:41.9017882Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-09-27T15:45:41.9018708Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-09-27T15:45:41.9020000Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:41.9020577Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-09-27T15:45:41.9023612Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-09-27T15:45:41.9024429Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-09-27T15:45:41.9025234Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-09-27T15:45:41.9145259Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-09-27T15:45:41.9190714Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-09-27T15:45:41.9191052Z    creating: build/lib/
2022-09-27T15:45:41.9192165Z   inflating: build/lib/libclog.a     
2022-09-27T15:45:41.9259173Z   inflating: build/lib/libgtest.a    
2022-09-27T15:45:41.9269618Z   inflating: build/lib/libpthreadpool.a  
2022-09-27T15:45:41.9361816Z   inflating: build/lib/libbenchmark.a  
2022-09-27T15:45:41.9469037Z   inflating: build/lib/libprotobuf-lite.a  
2022-09-27T15:45:41.9478276Z   inflating: build/lib/libittnotify.a  
2022-09-27T15:45:41.9510531Z   inflating: build/lib/libtensorpipe_uv.a  
2022-09-27T15:45:42.0049716Z   inflating: build/lib/libprotobuf.a  
2022-09-27T15:45:42.0127108Z   inflating: build/lib/libasmjit.a   
2022-09-27T15:45:42.0261510Z   inflating: build/lib/libgloo.a     
2022-09-27T15:45:42.0281354Z   inflating: build/lib/libfmt.a      
2022-09-27T15:45:42.0283203Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-09-27T15:45:42.0283807Z   inflating: build/lib/libfoxi_loader.a  
2022-09-27T15:45:42.0357599Z   inflating: build/lib/libc10.so     
2022-09-27T15:45:42.0358684Z   inflating: build/lib/libtorch_global_deps.so  
2022-09-27T15:45:42.0368835Z   inflating: build/lib/libcpuinfo.a  
2022-09-27T15:45:42.0378003Z   inflating: build/lib/libcpuinfo_internals.a  
2022-09-27T15:45:42.0394227Z   inflating: build/lib/libqnnpack.a  
2022-09-27T15:45:42.0418244Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-09-27T15:45:42.0993795Z   inflating: build/lib/libprotoc.a   
2022-09-27T15:45:42.0996364Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-09-27T15:45:42.1018892Z   inflating: build/lib/libnnpack.a   
2022-09-27T15:45:42.1039147Z   inflating: build/lib/libgmock.a    
2022-09-27T15:45:42.1039519Z   inflating: build/lib/libgtest_main.a  
2022-09-27T15:45:42.1040073Z   inflating: build/lib/libbenchmark_main.a  
2022-09-27T15:45:42.9250677Z   inflating: build/lib/libdnnl.a     
2022-09-27T15:45:42.9912302Z   inflating: build/lib/libtensorpipe.a  
2022-09-27T15:45:43.0055326Z   inflating: build/lib/libXNNPACK.a  
2022-09-27T15:45:43.0100768Z   inflating: build/lib/libc10_cuda.so  
2022-09-27T15:45:43.0101599Z   inflating: build/lib/libgmock_main.a  
2022-09-27T15:45:43.1647869Z   inflating: build/lib/libfbgemm.a   
2022-09-27T15:45:43.1940801Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-09-27T15:45:43.3083319Z   inflating: build/lib/libdnnl_graph.a  
2022-09-27T15:45:43.3508805Z   inflating: build/lib/libkineto.a   
2022-09-27T15:45:43.3554601Z   inflating: build/lib/libcaffe2_protos.a  
2022-09-27T15:45:43.3603266Z   inflating: build/lib/libonnx_proto.a  
2022-09-27T15:45:43.4288812Z   inflating: build/lib/libonnx.a     
2022-09-27T15:45:43.4723728Z   inflating: build/lib/libgloo_cuda.a  
2022-09-27T15:45:45.7981151Z   inflating: build/lib/libtorch_cpu.so  
2022-09-27T15:45:46.1437244Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-09-27T15:45:47.8565537Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-09-27T15:45:47.8566261Z   inflating: build/lib/libtorch_cuda.so  
2022-09-27T15:45:47.8567852Z   inflating: build/lib/libtorch.so   
2022-09-27T15:45:47.8571687Z   inflating: build/lib/libc10d_cuda_test.so  
2022-09-27T15:45:48.8546676Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-09-27T15:45:48.8570183Z   inflating: build/lib/libjitbackend_test.so  
2022-09-27T15:45:48.8631036Z   inflating: build/lib/libtorchbind_test.so  
2022-09-27T15:45:48.8661944Z   inflating: build/lib/libbackend_with_compiler.so  
2022-09-27T15:45:48.8666700Z   inflating: build/lib/libshm.so     
2022-09-27T15:45:49.0442138Z   inflating: build/lib/libtorch_python.so  
2022-09-27T15:45:49.0482170Z   inflating: build/lib/libnnapi_backend.so  
2022-09-27T15:45:49.0482488Z    creating: build/bin/
2022-09-27T15:45:49.0535786Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-09-27T15:45:49.0591377Z   inflating: build/bin/c10_DeviceGuard_test  
2022-09-27T15:45:49.0645460Z   inflating: build/bin/c10_Device_test  
2022-09-27T15:45:49.0709118Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-09-27T15:45:49.0760789Z   inflating: build/bin/c10_StreamGuard_test  
2022-09-27T15:45:49.0815049Z   inflating: build/bin/c10_SymInt_test  
2022-09-27T15:45:49.0875859Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-09-27T15:45:49.0936140Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-09-27T15:45:49.0997333Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-09-27T15:45:49.1050061Z   inflating: build/bin/c10_Array_test  
2022-09-27T15:45:49.1107278Z   inflating: build/bin/c10_Bitset_test  
2022-09-27T15:45:49.1162679Z   inflating: build/bin/c10_C++17_test  
2022-09-27T15:45:49.1214425Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-09-27T15:45:49.1267520Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-09-27T15:45:49.1321145Z   inflating: build/bin/c10_Half_test  
2022-09-27T15:45:49.1382226Z   inflating: build/bin/c10_LeftRight_test  
2022-09-27T15:45:49.1450119Z   inflating: build/bin/c10_Metaprogramming_test  
2022-09-27T15:45:49.1607733Z   inflating: build/bin/c10_SmallVectorTest  
2022-09-27T15:45:49.1661895Z   inflating: build/bin/c10_Synchronized_test  
2022-09-27T15:45:49.1723623Z   inflating: build/bin/c10_ThreadLocal_test  
2022-09-27T15:45:49.1780455Z   inflating: build/bin/c10_TypeIndex_test  
2022-09-27T15:45:49.1834974Z   inflating: build/bin/c10_TypeList_test  
2022-09-27T15:45:49.1886515Z   inflating: build/bin/c10_TypeTraits_test  
2022-09-27T15:45:49.1942100Z   inflating: build/bin/c10_accumulate_test  
2022-09-27T15:45:49.2002021Z   inflating: build/bin/c10_bfloat16_test  
2022-09-27T15:45:49.2060224Z   inflating: build/bin/c10_complex_math_test  
2022-09-27T15:45:49.2119963Z   inflating: build/bin/c10_complex_test  
2022-09-27T15:45:49.2238885Z   inflating: build/bin/c10_either_test  
2022-09-27T15:45:49.2295191Z   inflating: build/bin/c10_exception_test  
2022-09-27T15:45:49.2348970Z   inflating: build/bin/c10_flags_test  
2022-09-27T15:45:49.2533694Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-09-27T15:45:49.2587719Z   inflating: build/bin/c10_irange_test  
2022-09-27T15:45:49.2650105Z   inflating: build/bin/c10_logging_test  
2022-09-27T15:45:49.2730682Z   inflating: build/bin/c10_optional_test  
2022-09-27T15:45:49.2797562Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-09-27T15:45:49.2856981Z   inflating: build/bin/c10_registry_test  
2022-09-27T15:45:49.2923171Z   inflating: build/bin/c10_string_view_test  
2022-09-27T15:45:49.2984548Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-09-27T15:45:49.3039912Z   inflating: build/bin/c10_tempfile_test  
2022-09-27T15:45:49.3101516Z   inflating: build/bin/c10_typeid_test  
2022-09-27T15:45:49.3627735Z   inflating: build/bin/protoc-3.13.0.0  
2022-09-27T15:45:49.4153860Z   inflating: build/bin/protoc        
2022-09-27T15:45:49.4206044Z   inflating: build/bin/c10_cuda_CUDATest  
2022-09-27T15:45:49.4526325Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-09-27T15:45:49.4885613Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-09-27T15:45:49.4951002Z   inflating: build/bin/TCPStoreTest  
2022-09-27T15:45:49.5008708Z   inflating: build/bin/FileStoreTest  
2022-09-27T15:45:49.5066602Z   inflating: build/bin/HashStoreTest  
2022-09-27T15:45:49.5082416Z   inflating: build/bin/ProcessGroupMPITest  
2022-09-27T15:45:49.5085724Z   inflating: build/bin/example_allreduce  
2022-09-27T15:45:49.5142153Z   inflating: build/bin/Dimname_test  
2022-09-27T15:45:49.5220663Z   inflating: build/bin/Dict_test     
2022-09-27T15:45:49.5289389Z   inflating: build/bin/MaybeOwned_test  
2022-09-27T15:45:49.5351517Z   inflating: build/bin/NamedTensor_test  
2022-09-27T15:45:49.5415694Z   inflating: build/bin/apply_utils_test  
2022-09-27T15:45:49.5479240Z   inflating: build/bin/atest         
2022-09-27T15:45:49.5544439Z   inflating: build/bin/basic         
2022-09-27T15:45:49.5602252Z   inflating: build/bin/broadcast_test  
2022-09-27T15:45:49.5665426Z   inflating: build/bin/cpu_generator_test  
2022-09-27T15:45:49.5722207Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-09-27T15:45:49.5776126Z   inflating: build/bin/dispatch_key_set_test  
2022-09-27T15:45:49.5871280Z   inflating: build/bin/cpu_rng_test  
2022-09-27T15:45:49.5924733Z   inflating: build/bin/dlconvertor_test  
2022-09-27T15:45:49.5987406Z   inflating: build/bin/extension_backend_test  
2022-09-27T15:45:49.6047664Z   inflating: build/bin/half_test     
2022-09-27T15:45:49.6100588Z   inflating: build/bin/lazy_tensor_test  
2022-09-27T15:45:49.6201821Z   inflating: build/bin/ivalue_test   
2022-09-27T15:45:49.6258577Z   inflating: build/bin/memory_format_test  
2022-09-27T15:45:49.6315311Z   inflating: build/bin/math_kernel_test  
2022-09-27T15:45:49.6372276Z   inflating: build/bin/memory_overlapping_test  
2022-09-27T15:45:49.6426956Z   inflating: build/bin/operator_name_test  
2022-09-27T15:45:49.6487309Z   inflating: build/bin/native_test   
2022-09-27T15:45:49.6543658Z   inflating: build/bin/mobile_memory_cleanup  
2022-09-27T15:45:49.6597451Z   inflating: build/bin/operators_test  
2022-09-27T15:45:49.6654266Z   inflating: build/bin/packedtensoraccessor_test  
2022-09-27T15:45:49.6723778Z   inflating: build/bin/pow_test      
2022-09-27T15:45:49.6785938Z   inflating: build/bin/quantized_test  
2022-09-27T15:45:49.6840549Z   inflating: build/bin/reportMemoryUsage_test  
2022-09-27T15:45:49.6895519Z   inflating: build/bin/reduce_ops_test  
2022-09-27T15:45:49.6955606Z   inflating: build/bin/scalar_tensor_test  
2022-09-27T15:45:49.7017535Z   inflating: build/bin/scalar_test   
2022-09-27T15:45:49.7072782Z   inflating: build/bin/stride_properties_test  
2022-09-27T15:45:49.7157297Z   inflating: build/bin/tensor_iterator_test  
2022-09-27T15:45:49.7217430Z   inflating: build/bin/type_ptr_test  
2022-09-27T15:45:49.7220091Z   inflating: build/bin/thread_init_test  
2022-09-27T15:45:49.7279743Z   inflating: build/bin/test_parallel  
2022-09-27T15:45:49.7335919Z   inflating: build/bin/undefined_tensor_test  
2022-09-27T15:45:49.7400969Z   inflating: build/bin/type_test     
2022-09-27T15:45:49.7454233Z   inflating: build/bin/variant_test  
2022-09-27T15:45:49.7455760Z   inflating: build/bin/verify_api_visibility  
2022-09-27T15:45:49.7530554Z   inflating: build/bin/vmap_test     
2022-09-27T15:45:49.7584956Z   inflating: build/bin/weakref_test  
2022-09-27T15:45:49.7639594Z   inflating: build/bin/wrapdim_test  
2022-09-27T15:45:49.7704334Z   inflating: build/bin/IListRef_test  
2022-09-27T15:45:49.7757151Z   inflating: build/bin/xla_tensor_test  
2022-09-27T15:45:49.7876343Z   inflating: build/bin/List_test     
2022-09-27T15:45:49.8009249Z   inflating: build/bin/kernel_function_legacy_test  
2022-09-27T15:45:49.8078894Z   inflating: build/bin/KernelFunction_test  
2022-09-27T15:45:49.8183326Z   inflating: build/bin/kernel_function_test  
2022-09-27T15:45:49.8322991Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-09-27T15:45:49.8436701Z   inflating: build/bin/kernel_lambda_test  
2022-09-27T15:45:49.8501472Z   inflating: build/bin/kernel_stackbased_test  
2022-09-27T15:45:49.8555771Z   inflating: build/bin/CppSignature_test  
2022-09-27T15:45:49.8660319Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-09-27T15:45:49.8711823Z   inflating: build/bin/op_allowlist_test  
2022-09-27T15:45:49.8768803Z   inflating: build/bin/inline_container_test  
2022-09-27T15:45:49.9087219Z   inflating: build/bin/op_registration_test  
2022-09-27T15:45:49.9148297Z   inflating: build/bin/backend_fallback_test  
2022-09-27T15:45:49.9204299Z   inflating: build/bin/cuda_apply_test  
2022-09-27T15:45:49.9270861Z   inflating: build/bin/cuda_atomic_ops_test  
2022-09-27T15:45:49.9328374Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-09-27T15:45:49.9381681Z   inflating: build/bin/cuda_device_test  
2022-09-27T15:45:49.9455604Z   inflating: build/bin/cuda_complex_math_test  
2022-09-27T15:45:49.9518881Z   inflating: build/bin/cuda_complex_test  
2022-09-27T15:45:49.9583680Z   inflating: build/bin/cuda_cub_test  
2022-09-27T15:45:49.9637321Z   inflating: build/bin/cuda_dlconvertor_test  
2022-09-27T15:45:49.9692810Z   inflating: build/bin/cuda_integer_divider_test  
2022-09-27T15:45:49.9765583Z   inflating: build/bin/cuda_distributions_test  
2022-09-27T15:45:49.9829062Z   inflating: build/bin/cuda_generator_test  
2022-09-27T15:45:49.9882906Z   inflating: build/bin/cuda_half_test  
2022-09-27T15:45:49.9935428Z   inflating: build/bin/cuda_optional_test  
2022-09-27T15:45:50.0001656Z   inflating: build/bin/cuda_stream_test  
2022-09-27T15:45:50.0058201Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-09-27T15:45:50.0113911Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-09-27T15:45:50.0166230Z   inflating: build/bin/cuda_cudnn_test  
2022-09-27T15:45:50.0223586Z   inflating: build/bin/cuda_vectorized_test  
2022-09-27T15:45:50.0240757Z   inflating: build/bin/tutorial_tensorexpr  
2022-09-27T15:45:50.0312232Z   inflating: build/bin/ProcessGroupGlooTest  
2022-09-27T15:45:50.0375362Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-09-27T15:45:50.0442516Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-09-27T15:45:50.0506281Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-09-27T15:45:50.0563404Z   inflating: build/bin/ProcessGroupUCCTest  
2022-09-27T15:45:50.0621682Z   inflating: build/bin/test_dist_autograd  
2022-09-27T15:45:50.0696894Z   inflating: build/bin/test_cpp_rpc  
2022-09-27T15:45:50.0699739Z   inflating: build/bin/parallel_benchmark  
2022-09-27T15:45:50.0774529Z   inflating: build/bin/test_mobile_nnc  
2022-09-27T15:45:50.0785852Z   inflating: build/bin/aot_model_compiler_test  
2022-09-27T15:45:50.1718075Z   inflating: build/bin/test_tensorexpr  
2022-09-27T15:45:50.2100709Z   inflating: build/bin/test_lazy     
2022-09-27T15:45:50.2106401Z   inflating: build/bin/torch_shm_manager  
2022-09-27T15:45:50.3427399Z   inflating: build/bin/test_api      
2022-09-27T15:45:50.4576167Z   inflating: build/bin/test_jit      
2022-09-27T15:45:50.4578061Z   inflating: .pytorch-test-times.json  
2022-09-27T15:45:50.4613046Z ##[group]Run df -H
2022-09-27T15:45:50.4613320Z df -H
2022-09-27T15:45:50.4626775Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:45:50.4627092Z env:
2022-09-27T15:45:50.4627354Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:50.4627612Z   GPU_FLAG: --gpus all
2022-09-27T15:45:50.4627895Z ##[endgroup]
2022-09-27T15:45:50.4668339Z Filesystem      Size  Used Avail Use% Mounted on
2022-09-27T15:45:50.4668675Z devtmpfs        129G     0  129G   0% /dev
2022-09-27T15:45:50.4668981Z tmpfs           129G     0  129G   0% /dev/shm
2022-09-27T15:45:50.4669220Z tmpfs           129G  549k  129G   1% /run
2022-09-27T15:45:50.4671976Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-09-27T15:45:50.4672561Z /dev/xvda1      162G   30G  132G  19% /
2022-09-27T15:45:50.4698245Z ##[group]Run .github/scripts/parse_ref.py
2022-09-27T15:45:50.4698617Z .github/scripts/parse_ref.py
2022-09-27T15:45:50.4710785Z shell: /usr/bin/bash -e {0}
2022-09-27T15:45:50.4711054Z env:
2022-09-27T15:45:50.4711311Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:50.4711571Z   GPU_FLAG: --gpus all
2022-09-27T15:45:50.4711834Z ##[endgroup]
2022-09-27T15:45:50.5039391Z ##[group]Run set -x
2022-09-27T15:45:50.5039816Z set -x
2022-09-27T15:45:50.5040068Z 
2022-09-27T15:45:50.5040371Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-09-27T15:45:50.5040714Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-09-27T15:45:50.5041081Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-09-27T15:45:50.5041424Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-09-27T15:45:50.5041685Z else
2022-09-27T15:45:50.5041974Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:45:50.5042266Z fi
2022-09-27T15:45:50.5042475Z 
2022-09-27T15:45:50.5042805Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-09-27T15:45:50.5043133Z 
2022-09-27T15:45:50.5043443Z # sanitize the input commit message and PR body here:
2022-09-27T15:45:50.5043729Z #
2022-09-27T15:45:50.5044138Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-09-27T15:45:50.5044665Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-09-27T15:45:50.5045085Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-09-27T15:45:50.5045414Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-09-27T15:45:50.5045685Z 
2022-09-27T15:45:50.5046030Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-09-27T15:45:50.5046425Z # wreak havoc internally
2022-09-27T15:45:50.5046760Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-09-27T15:45:50.5047100Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-09-27T15:45:50.5047350Z 
2022-09-27T15:45:50.5047674Z # detached container should get cleaned up by teardown_ec2_linux
2022-09-27T15:45:50.5048091Z # TODO: Stop building test binaries as part of the build phase
2022-09-27T15:45:50.5048457Z # Used for GPU_FLAG since that doesn't play nice
2022-09-27T15:45:50.5048801Z # shellcheck disable=SC2086,SC2090
2022-09-27T15:45:50.5049271Z container_name=$(docker run \
2022-09-27T15:45:50.5049540Z   ${GPU_FLAG:-} \
2022-09-27T15:45:50.5050079Z   -e BUILD_ENVIRONMENT \
2022-09-27T15:45:50.5050380Z   -e PR_NUMBER \
2022-09-27T15:45:50.5050671Z   -e GITHUB_ACTIONS \
2022-09-27T15:45:50.5050927Z   -e BASE_SHA \
2022-09-27T15:45:50.5051198Z   -e BRANCH \
2022-09-27T15:45:50.5051461Z   -e SHA1 \
2022-09-27T15:45:50.5051713Z   -e AWS_DEFAULT_REGION \
2022-09-27T15:45:50.5051996Z   -e IN_WHEEL_TEST \
2022-09-27T15:45:50.5052271Z   -e SHARD_NUMBER \
2022-09-27T15:45:50.5052518Z   -e TEST_CONFIG \
2022-09-27T15:45:50.5052794Z   -e NUM_TEST_SHARDS \
2022-09-27T15:45:50.5053066Z   -e PR_BODY \
2022-09-27T15:45:50.5053327Z   -e COMMIT_MESSAGES \
2022-09-27T15:45:50.5053631Z   -e PYTORCH_RETRY_TEST_CASES \
2022-09-27T15:45:50.5053960Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-09-27T15:45:50.5054240Z   -e PR_LABELS \
2022-09-27T15:45:50.5054535Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-09-27T15:45:50.5054838Z   -e SCCACHE_BUCKET \
2022-09-27T15:45:50.5055132Z   -e SCCACHE_S3_KEY_PREFIX \
2022-09-27T15:45:50.5055392Z   -e XLA_CUDA \
2022-09-27T15:45:50.5055686Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-09-27T15:45:50.5056040Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-09-27T15:45:50.5056352Z   --ulimit stack=10485760:83886080 \
2022-09-27T15:45:50.5056673Z   --security-opt seccomp=unconfined \
2022-09-27T15:45:50.5056991Z   --cap-add=SYS_PTRACE \
2022-09-27T15:45:50.5057249Z   --ipc=host \
2022-09-27T15:45:50.5057622Z   --shm-size="${SHM_SIZE}" \
2022-09-27T15:45:50.5057915Z   --tty \
2022-09-27T15:45:50.5058147Z   --detach \
2022-09-27T15:45:50.5058432Z   --name="${container_name}" \
2022-09-27T15:45:50.5058723Z   --user jenkins \
2022-09-27T15:45:50.5059055Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-09-27T15:45:50.5059386Z   -w /var/lib/jenkins/workspace \
2022-09-27T15:45:50.5059681Z   "${DOCKER_IMAGE}"
2022-09-27T15:45:50.5059931Z )
2022-09-27T15:45:50.5060261Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-09-27T15:45:50.5072272Z shell: /usr/bin/bash -e {0}
2022-09-27T15:45:50.5072543Z env:
2022-09-27T15:45:50.5072776Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:50.5073061Z   GPU_FLAG: --gpus all
2022-09-27T15:45:50.5073406Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:45:50.5073712Z   PR_NUMBER: 85462
2022-09-27T15:45:50.5073977Z   BRANCH: pull/85462
2022-09-27T15:45:50.5074283Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:45:50.5074597Z   BASE_SHA: 76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:45:50.5074919Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T15:45:50.5075222Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T15:45:50.5075494Z   TEST_CONFIG: distributed
2022-09-27T15:45:50.5075765Z   SHARD_NUMBER: 3
2022-09-27T15:45:50.5076025Z   NUM_TEST_SHARDS: 3
2022-09-27T15:45:50.5078291Z   PR_BODY: Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.

I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. 

Here is the code to reproduce the bug:

```
import torch
import weakref
from transformers import DetrForObjectDetection

class X:
    def __init__(self, optimizer):
        self.optimizer = optimizer

        # Will cause cyclic reference.
        self.func = self.dummy

        # Will work as expected, memory cleared after instance count is zero.
        # self.func = weakref.WeakMethod(self.dummy)

    def dummy(self, x):
        return 1.

def test():
    model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50')
    model.to('cuda')
    optimizer = torch.optim.Adam(model.parameters())
    x = X(optimizer)

test()
print(f'{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).
```
2022-09-27T15:45:50.5080877Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-09-27T15:45:50.5081231Z   SCCACHE_S3_KEY_PREFIX: pull
2022-09-27T15:45:50.5081505Z   SHM_SIZE: 2g
2022-09-27T15:45:50.5081991Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:45:50.5082487Z   XLA_CUDA: 
2022-09-27T15:45:50.5082852Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:45:50.5083199Z ##[endgroup]
2022-09-27T15:45:50.5112180Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-09-27T15:45:50.5112678Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-09-27T15:45:50.5113015Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:45:50.5116449Z ++ git cherry -v origin/master
2022-09-27T15:45:50.5363804Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR
2022-09-27T15:45:50.5364318Z + acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR
2022-09-27T15:45:50.5364705Z + 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form
2022-09-27T15:45:50.5365124Z + 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference
2022-09-27T15:45:50.5365509Z + 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:45:50.5366831Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:45:50.5383548Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained('\''facebook/detr-resnet-50'\'')    model.to('\''cuda'\'')    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f'\''{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}'\'')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:45:50.5386872Z + export 'COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:45:50.5388314Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:45:50.5395589Z + export 'PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:45:50.5401082Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:45:50.5403516Z +++ nproc --ignore=2
2022-09-27T15:45:50.5442863Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3133193930 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:46:04.2306890Z + container_name=4d347b8a91a0d81ea51e22a5c6247010f4ea86e52f35fb357941dd928b7dc4b6
2022-09-27T15:46:04.2307890Z + docker exec -t 4d347b8a91a0d81ea51e22a5c6247010f4ea86e52f35fb357941dd928b7dc4b6 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-09-27T15:46:04.8201669Z Processing ./dist/torch-1.13.0a0+git52424e2-cp310-cp310-linux_x86_64.whl
2022-09-27T15:46:04.9140377Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+git52424e2) (4.3.0)
2022-09-27T15:46:05.8726961Z Installing collected packages: torch
2022-09-27T15:46:16.6902448Z Successfully installed torch-1.13.0a0+git52424e2
2022-09-27T15:46:16.8702713Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-09-27T15:46:16.8927240Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-09-27T15:46:16.8932360Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T15:46:16.8932860Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-09-27T15:46:16.8933513Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-09-27T15:46:16.8934009Z + BUILD_DIR=build
2022-09-27T15:46:16.8934297Z + BUILD_RENAMED_DIR=build_renamed
2022-09-27T15:46:16.8934559Z + BUILD_BIN_DIR=build/bin
2022-09-27T15:46:16.8934838Z + export VALGRIND=ON
2022-09-27T15:46:16.8936748Z + VALGRIND=ON
2022-09-27T15:46:16.8937339Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-09-27T15:46:16.8937831Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-09-27T15:46:16.8938087Z ++ realpath build/custom_test_artifacts
2022-09-27T15:46:16.8942372Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-09-27T15:46:16.8946189Z ++ dirname .jenkins/pytorch/test.sh
2022-09-27T15:46:16.8954269Z + source .jenkins/pytorch/common.sh
2022-09-27T15:46:16.8958022Z +++ dirname .jenkins/pytorch/common.sh
2022-09-27T15:46:16.8968512Z ++ source .jenkins/pytorch/common_utils.sh
2022-09-27T15:46:16.8970664Z +++ declare -f -t trap_add
2022-09-27T15:46:16.8974990Z ++ set -ex
2022-09-27T15:46:16.8975443Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:16.8975718Z ++ BUILD_TEST_LIBTORCH=0
2022-09-27T15:46:16.8977211Z ++ [[ distributed == *xla* ]]
2022-09-27T15:46:16.8977803Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-09-27T15:46:16.8978295Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *linux-bionic* ]]
2022-09-27T15:46:16.8978636Z ++ which conda
2022-09-27T15:46:16.8987419Z /opt/conda/bin/conda
2022-09-27T15:46:16.8988191Z ++ conda install -q -y cmake
2022-09-27T15:46:19.5898948Z Collecting package metadata (current_repodata.json): ...working... done
2022-09-27T15:46:20.3429705Z Solving environment: ...working... done
2022-09-27T15:46:20.4482823Z 
2022-09-27T15:46:20.4483117Z ## Package Plan ##
2022-09-27T15:46:20.4483361Z 
2022-09-27T15:46:20.4483951Z   environment location: /opt/conda
2022-09-27T15:46:20.4484168Z 
2022-09-27T15:46:20.4484306Z   added / updated specs:
2022-09-27T15:46:20.4484731Z     - cmake
2022-09-27T15:46:20.4484894Z 
2022-09-27T15:46:20.4484930Z 
2022-09-27T15:46:20.4485096Z The following packages will be downloaded:
2022-09-27T15:46:20.4485305Z 
2022-09-27T15:46:20.4485451Z     package                    |            build
2022-09-27T15:46:20.4485813Z     ---------------------------|-----------------
2022-09-27T15:46:20.4487034Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-09-27T15:46:20.4487500Z     certifi-2022.9.14          |  py310h06a4308_0         155 KB
2022-09-27T15:46:20.4487902Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-09-27T15:46:20.4488330Z     conda-22.9.0               |  py310h06a4308_0         894 KB
2022-09-27T15:46:20.4488733Z     expat-2.4.4                |       h295c915_0         169 KB
2022-09-27T15:46:20.4489129Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-09-27T15:46:20.4489507Z     libcurl-7.82.0             |       h0b77cf5_0         342 KB
2022-09-27T15:46:20.4490241Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-09-27T15:46:20.4490648Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-09-27T15:46:20.4491042Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-09-27T15:46:20.4491452Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-09-27T15:46:20.4491849Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-09-27T15:46:20.4492478Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-09-27T15:46:20.4492898Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-09-27T15:46:20.4493298Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-09-27T15:46:20.4493739Z     ------------------------------------------------------------
2022-09-27T15:46:20.4494067Z                                            Total:        12.9 MB
2022-09-27T15:46:20.4494246Z 
2022-09-27T15:46:20.4494413Z The following NEW packages will be INSTALLED:
2022-09-27T15:46:20.4494621Z 
2022-09-27T15:46:20.4495000Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-09-27T15:46:20.4495491Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-09-27T15:46:20.4495954Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-09-27T15:46:20.4496425Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-09-27T15:46:20.4496907Z   libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
2022-09-27T15:46:20.4497393Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-09-27T15:46:20.4497894Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-09-27T15:46:20.4498402Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-09-27T15:46:20.4498908Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-09-27T15:46:20.4499365Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-09-27T15:46:20.4499839Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-09-27T15:46:20.4500309Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-09-27T15:46:20.4500755Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-09-27T15:46:20.4500966Z 
2022-09-27T15:46:20.4501115Z The following packages will be UPDATED:
2022-09-27T15:46:20.4501312Z 
2022-09-27T15:46:20.4501600Z   certifi                         2022.6.15-py310h06a4308_0 --> 2022.9.14-py310h06a4308_0
2022-09-27T15:46:20.4502077Z   conda                              4.14.0-py310h06a4308_0 --> 22.9.0-py310h06a4308_0
2022-09-27T15:46:20.4502448Z 
2022-09-27T15:46:20.4502466Z 
2022-09-27T15:46:21.8755375Z Preparing transaction: ...working... done
2022-09-27T15:46:22.4035125Z Verifying transaction: ...working... done
2022-09-27T15:46:23.8817767Z Executing transaction: ...working... done
2022-09-27T15:46:24.0191864Z Retrieving notices: ...working... done
2022-09-27T15:46:24.2773554Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-09-27T15:46:24.2773992Z + echo 'Environment variables'
2022-09-27T15:46:24.2774295Z Environment variables
2022-09-27T15:46:24.2774554Z + env
2022-09-27T15:46:24.2783633Z SHARD_NUMBER=3
2022-09-27T15:46:24.2784471Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-09-27T15:46:24.2784885Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-09-27T15:46:24.2785274Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-09-27T15:46:24.2785878Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-09-27T15:46:24.2786204Z UCC_HOME=/usr
2022-09-27T15:46:24.2786589Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:46:24.2787058Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-09-27T15:46:24.2787357Z INSTALLED_DB=yes
2022-09-27T15:46:24.2787626Z HOSTNAME=4d347b8a91a0
2022-09-27T15:46:24.2787905Z GITHUB_REF_NAME=85462/merge
2022-09-27T15:46:24.2788222Z GITHUB_API_URL=https://api.github.com
2022-09-27T15:46:24.2788543Z OPENSSL_DIR=/opt/openssl
2022-09-27T15:46:24.2788902Z UCC_COMMIT=12944da33f911daf505d9bbc51411233d0ed85e1
2022-09-27T15:46:24.2789621Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_47546a13-fdaf-4460-9574-4d2097995603
2022-09-27T15:46:24.2790027Z CUDA_PATH=/usr/local/cuda
2022-09-27T15:46:24.2790562Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-09-27T15:46:24.2790950Z GITHUB_RUN_ATTEMPT=2
2022-09-27T15:46:24.2791579Z TEST_CONFIG=distributed
2022-09-27T15:46:24.2791956Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-09-27T15:46:24.2792464Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-09-27T15:46:24.2792845Z GITHUB_REPOSITORY_OWNER=pytorch
2022-09-27T15:46:24.2793210Z GITHUB_ACTIONS=true
2022-09-27T15:46:24.2793498Z NVIDIA_VISIBLE_DEVICES=all
2022-09-27T15:46:24.2793795Z NV_NVPROF_VERSION=11.6.124-1
2022-09-27T15:46:24.2794116Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-09-27T15:46:24.2794400Z CI=true
2022-09-27T15:46:24.2794658Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-09-27T15:46:24.2795070Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-09-27T15:46:24.2795389Z BRANCH=pull/85462
2022-09-27T15:46:24.2795722Z GITHUB_HEAD_REF=cycliclr-memory-fix
2022-09-27T15:46:24.2796060Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-09-27T15:46:24.2796381Z GITHUB_ACTOR=kongzii
2022-09-27T15:46:24.2796693Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-09-27T15:46:24.2796975Z GITHUB_ACTION_REF=
2022-09-27T15:46:24.2797271Z NCCL_VERSION=2.12.10-1
2022-09-27T15:46:24.2797546Z GITHUB_ACTION=__self
2022-09-27T15:46:24.2797777Z VALGRIND=ON
2022-09-27T15:46:24.2798048Z GITHUB_REF_PROTECTED=false
2022-09-27T15:46:24.2799190Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:46:24.2800108Z ***
2022-09-27T15:46:24.2800344Z INSTALLED_VISION=yes
2022-09-27T15:46:24.2800605Z NVARCH=x86_64
2022-09-27T15:46:24.2800933Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-09-27T15:46:24.2801204Z HOME=/var/lib/jenkins
2022-09-27T15:46:24.2801487Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-09-27T15:46:24.2801781Z GITHUB_ACTION_REPOSITORY=
2022-09-27T15:46:24.2802035Z GITHUB_REF_TYPE=branch
2022-09-27T15:46:24.2802359Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-09-27T15:46:24.2802659Z GITHUB_RETENTION_DAYS=90
2022-09-27T15:46:24.2803033Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-09-27T15:46:24.2803453Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-09-27T15:46:24.2804018Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_47546a13-fdaf-4460-9574-4d2097995603
2022-09-27T15:46:24.2804437Z DEBIAN_FRONTEND=noninteractive
2022-09-27T15:46:24.2804942Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-09-27T15:46:24.2805262Z GITHUB_REF=refs/pull/85462/merge
2022-09-27T15:46:24.2805580Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-09-27T15:46:24.2805885Z GITHUB_SHA=1faa2af6dbb8dd899ab20874e9966185467c5883
2022-09-27T15:46:24.2806202Z INSTALLED_PROTOBUF=yes
2022-09-27T15:46:24.2806478Z GITHUB_RUN_ID=3133193930
2022-09-27T15:46:24.2806815Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-09-27T15:46:24.2807129Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-09-27T15:46:24.2807441Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-09-27T15:46:24.2807744Z NV_NVTX_VERSION=11.6.124-1
2022-09-27T15:46:24.2808059Z GITHUB_SERVER_URL=https://github.com
2022-09-27T15:46:24.2808349Z MAX_JOBS=30
2022-09-27T15:46:24.2808627Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-09-27T15:46:24.2809023Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-09-27T15:46:24.2809527Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-09-27T15:46:24.2810356Z UCX_HOME=/usr
2022-09-27T15:46:24.2810637Z PYTORCH_RETRY_TEST_CASES=1
2022-09-27T15:46:24.2810989Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-09-27T15:46:24.2811329Z BASE_SHA=76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:46:24.2811699Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-09-27T15:46:24.2815259Z PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```
2022-09-27T15:46:24.2817696Z GITHUB_BASE_REF=master
2022-09-27T15:46:24.2817938Z TERM=xterm
2022-09-27T15:46:24.2818178Z XLA_CUDA=
2022-09-27T15:46:24.2818473Z NV_NVML_DEV_VERSION=11.6.55-1
2022-09-27T15:46:24.2818741Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-09-27T15:46:24.2819018Z CUDA_VERSION=11.6.2
2022-09-27T15:46:24.2819378Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-09-27T15:46:24.2819673Z OPENSSL_ROOT_DIR=/opt/openssl
2022-09-27T15:46:24.2820220Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_47546a13-fdaf-4460-9574-4d2097995603
2022-09-27T15:46:24.2820615Z GITHUB_JOB=test
2022-09-27T15:46:24.2820865Z SCCACHE_S3_KEY_PREFIX=pull
2022-09-27T15:46:24.2821585Z COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting
2022-09-27T15:46:24.2822371Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-09-27T15:46:24.2822678Z NUM_TEST_SHARDS=3
2022-09-27T15:46:24.2822916Z PR_NUMBER=85462
2022-09-27T15:46:24.2823163Z SHLVL=1
2022-09-27T15:46:24.2823530Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-09-27T15:46:24.2823846Z GITHUB_REPOSITORY=pytorch/pytorch
2022-09-27T15:46:24.2824458Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471
2022-09-27T15:46:24.2825056Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-09-27T15:46:24.2825378Z SHA1=52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:46:24.2825661Z GITHUB_EVENT_NAME=pull_request
2022-09-27T15:46:24.2825995Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-09-27T15:46:24.2826360Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-09-27T15:46:24.2826638Z GITHUB_RUN_NUMBER=50832
2022-09-27T15:46:24.2826920Z GITHUB_WORKFLOW=pull
2022-09-27T15:46:24.2827352Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:24.2827815Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-09-27T15:46:24.2828248Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:46:24.2828613Z GITHUB_TRIGGERING_ACTOR=albanD
2022-09-27T15:46:24.2828885Z _=/usr/bin/env
2022-09-27T15:46:24.2829164Z + echo 'Testing pytorch'
2022-09-27T15:46:24.2829438Z Testing pytorch
2022-09-27T15:46:24.2829722Z + export LANG=C.UTF-8
2022-09-27T15:46:24.2829976Z + LANG=C.UTF-8
2022-09-27T15:46:24.2830228Z + PR_NUMBER=85462
2022-09-27T15:46:24.2830509Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-09-27T15:46:24.2830861Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-09-27T15:46:24.2831292Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:24.2831630Z + [[ distributed == \s\l\o\w ]]
2022-09-27T15:46:24.2832036Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-09-27T15:46:24.2832498Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-09-27T15:46:24.2832857Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-09-27T15:46:24.2833161Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-09-27T15:46:24.2833582Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda11* ]]
2022-09-27T15:46:24.2833912Z + export BUILD_SPLIT_CUDA=ON
2022-09-27T15:46:24.2834168Z + BUILD_SPLIT_CUDA=ON
2022-09-27T15:46:24.2834452Z + [[ distributed == *crossref* ]]
2022-09-27T15:46:24.2834737Z + [[ distributed == *dynamo* ]]
2022-09-27T15:46:24.2835043Z + [[ -n 85462 ]]
2022-09-27T15:46:24.2835291Z + [[ -z '' ]]
2022-09-27T15:46:24.2835590Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:46:24.2835932Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:46:24.2836322Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:24.2836770Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-09-27T15:46:24.2837145Z + pip_install --user ninja
2022-09-27T15:46:24.2837497Z + pip install --progress-bar off --user ninja
2022-09-27T15:46:24.8611722Z Collecting ninja
2022-09-27T15:46:24.8813609Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-09-27T15:46:25.7701520Z Installing collected packages: ninja
2022-09-27T15:46:25.7942014Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-09-27T15:46:25.7942701Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T15:46:25.8015525Z Successfully installed ninja-1.10.2.3
2022-09-27T15:46:25.8753065Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:25.8753723Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:25.8754931Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-09-27T15:46:25.8755304Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-09-27T15:46:25.8755643Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-09-27T15:46:25.8759268Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-09-27T15:46:25.8775292Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-09-27T15:46:25.8775776Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-09-27T15:46:25.8778871Z + cd test
2022-09-27T15:46:25.8779391Z + python -c 'import torch; print(torch.__config__.show())'
2022-09-27T15:46:27.5550971Z PyTorch built with:
2022-09-27T15:46:27.5551977Z   - GCC 7.5
2022-09-27T15:46:27.5552481Z   - C++ Version: 201402
2022-09-27T15:46:27.5553048Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:46:27.5553630Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:46:27.5554053Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:46:27.5554443Z   - LAPACK is enabled (usually provided by MKL)
2022-09-27T15:46:27.5554786Z   - NNPACK is enabled
2022-09-27T15:46:27.5555090Z   - CPU capability usage: AVX2
2022-09-27T15:46:27.5555406Z   - CUDA Runtime 11.6
2022-09-27T15:46:27.5555813Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-09-27T15:46:27.5556196Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-09-27T15:46:27.5556518Z   - Magma 2.6.1
2022-09-27T15:46:27.5559734Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-09-27T15:46:27.5561846Z 
2022-09-27T15:46:27.8138550Z + cd test
2022-09-27T15:46:27.8139491Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-09-27T15:46:29.4415192Z ATen/Parallel:
2022-09-27T15:46:29.4415855Z 	at::get_num_threads() : 16
2022-09-27T15:46:29.4416487Z 	at::get_num_interop_threads() : 16
2022-09-27T15:46:29.4416882Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:46:29.4417152Z 	omp_get_max_threads() : 16
2022-09-27T15:46:29.4417849Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:46:29.4418257Z 	mkl_get_max_threads() : 16
2022-09-27T15:46:29.4418714Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:46:29.4419075Z std::thread::hardware_concurrency() : 32
2022-09-27T15:46:29.4419381Z Environment variables:
2022-09-27T15:46:29.4419663Z 	OMP_NUM_THREADS : [not set]
2022-09-27T15:46:29.4419924Z 	MKL_NUM_THREADS : [not set]
2022-09-27T15:46:29.4420243Z ATen parallel backend: OpenMP
2022-09-27T15:46:29.4420427Z 
2022-09-27T15:46:29.6871736Z + [[ distributed == *deploy* ]]
2022-09-27T15:46:29.6872492Z + [[ distributed == *backward* ]]
2022-09-27T15:46:29.6873255Z + [[ distributed == *xla* ]]
2022-09-27T15:46:29.6873566Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-09-27T15:46:29.6874154Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-09-27T15:46:29.6874541Z + [[ distributed == distributed ]]
2022-09-27T15:46:29.6874849Z + install_torchdynamo
2022-09-27T15:46:29.6875012Z + local commit
2022-09-27T15:46:29.6875536Z ++ get_pinned_commit torchdynamo
2022-09-27T15:46:29.6875978Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-09-27T15:46:29.6892232Z + commit=41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:29.6893404Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:29.6894150Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:30.1814301Z Collecting git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:30.1820204Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 41c44bc1d080d6cf063419a4166732b983b84eef) to /tmp/pip-req-build-_zfc72qr
2022-09-27T15:46:30.1841317Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-_zfc72qr
2022-09-27T15:46:31.0040138Z   Running command git rev-parse -q --verify 'sha^41c44bc1d080d6cf063419a4166732b983b84eef'
2022-09-27T15:46:31.0062220Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:31.3371713Z   Running command git checkout -q 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:31.6605414Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:34.2220956Z   Preparing metadata (setup.py) ... [?25l- done
2022-09-27T15:46:34.2293646Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+git52424e2)
2022-09-27T15:46:34.2297983Z Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.21.2)
2022-09-27T15:46:34.2737612Z Collecting tabulate
2022-09-27T15:46:34.2984473Z   Downloading tabulate-0.8.10-py3-none-any.whl (29 kB)
2022-09-27T15:46:34.3050527Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from torchdynamo==1.13.0.dev0) (6.0)
2022-09-27T15:46:34.3056121Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.11.1)
2022-09-27T15:46:34.3082443Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-09-27T15:46:34.3116738Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-09-27T15:46:34.3249805Z Building wheels for collected packages: torchdynamo
2022-09-27T15:46:39.0940972Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-09-27T15:46:39.1042291Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp310-cp310-linux_x86_64.whl size=2600957 sha256=d67bdd49ce1f6124cac3354d24337404c3c12fb1733757f1537b08bfe3fc6da8
2022-09-27T15:46:39.1044246Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/2e/47/4b/a72e6a8c4801cae81c62fd871ce3601d87ba0b7e2d5534e15c
2022-09-27T15:46:39.1073331Z Successfully built torchdynamo
2022-09-27T15:46:40.0053386Z Installing collected packages: tabulate, torchdynamo
2022-09-27T15:46:42.5610653Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0
2022-09-27T15:46:42.6632285Z + test_distributed
2022-09-27T15:46:42.6632851Z + echo 'Testing distributed python tests'
2022-09-27T15:46:42.6633155Z Testing distributed python tests
2022-09-27T15:46:42.6633560Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose
2022-09-27T15:46:44.9538246Z Ignoring disabled issues:  []
2022-09-27T15:46:44.9909964Z /var/lib/jenkins/workspace/test/run_test.py:960: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-09-27T15:46:44.9910551Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-09-27T15:46:44.9912585Z Found test time stats from artifacts
2022-09-27T15:46:44.9914461Z Selected tests:
2022-09-27T15:46:44.9914721Z  distributed/test_c10d_nccl
2022-09-27T15:46:44.9915021Z  distributed/test_c10d_gloo
2022-09-27T15:46:44.9915351Z  distributed/fsdp/test_fsdp_mixed_precision
2022-09-27T15:46:44.9915685Z  distributed/fsdp/test_fsdp_summon_full_params
2022-09-27T15:46:44.9916066Z  distributed/optim/test_zero_redundancy_optimizer
2022-09-27T15:46:44.9916442Z  distributed/fsdp/test_fsdp_comm_hooks
2022-09-27T15:46:44.9916749Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-09-27T15:46:44.9917116Z  distributed/algorithms/quantization/test_quantization
2022-09-27T15:46:44.9917473Z  distributed/test_pg_wrapper
2022-09-27T15:46:44.9917769Z  distributed/test_store
2022-09-27T15:46:44.9918032Z  distributed/algorithms/test_join
2022-09-27T15:46:44.9918340Z  distributed/test_c10d_spawn_gloo
2022-09-27T15:46:44.9918677Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-09-27T15:46:44.9919024Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-09-27T15:46:44.9919350Z  distributed/test_c10d_common
2022-09-27T15:46:44.9919682Z  distributed/elastic/multiprocessing/api_test
2022-09-27T15:46:44.9920520Z  distributed/_shard/checkpoint/test_checkpoint
2022-09-27T15:46:44.9920927Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-09-27T15:46:44.9921309Z  distributed/test_c10d_object_collectives
2022-09-27T15:46:44.9921663Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-09-27T15:46:44.9922174Z  distributed/fsdp/test_fsdp_overlap
2022-09-27T15:46:44.9922563Z  distributed/_shard/checkpoint/test_file_system_checkpoint_cpu
2022-09-27T15:46:44.9922987Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-09-27T15:46:44.9923317Z  distributed/elastic/timer/local_timer_example
2022-09-27T15:46:44.9923688Z  distributed/fsdp/test_distributed_checkpoint
2022-09-27T15:46:44.9924053Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-09-27T15:46:44.9924437Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-09-27T15:46:44.9924781Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-09-27T15:46:44.9925133Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-09-27T15:46:44.9925477Z  distributed/fsdp/test_fsdp_uneven
2022-09-27T15:46:44.9925815Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-09-27T15:46:44.9926122Z  distributed/fsdp/test_fsdp_fx
2022-09-27T15:46:44.9926415Z  distributed/test_data_parallel
2022-09-27T15:46:44.9926736Z  distributed/rpc/test_share_memory
2022-09-27T15:46:44.9927095Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-09-27T15:46:44.9927454Z  distributed/elastic/utils/util_test
2022-09-27T15:46:44.9927754Z  distributed/nn/jit/test_instantiator
2022-09-27T15:46:44.9928085Z  distributed/_shard/checkpoint/test_planner
2022-09-27T15:46:44.9928398Z  distributed/fsdp/test_utils
2022-09-27T15:46:44.9928690Z  distributed/elastic/metrics/api_test
2022-09-27T15:46:44.9928998Z  distributed/_shard/test_replicated_tensor
2022-09-27T15:46:44.9929324Z  distributed/elastic/timer/api_test
2022-09-27T15:46:44.9930029Z  distributed/pipeline/sync/skip/test_gpipe
2022-09-27T15:46:44.9930370Z  distributed/pipeline/sync/skip/test_portal
2022-09-27T15:46:44.9930743Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-09-27T15:46:44.9931108Z  distributed/pipeline/sync/test_checkpoint
2022-09-27T15:46:44.9931426Z  distributed/pipeline/sync/test_dependency
2022-09-27T15:46:44.9931775Z  distributed/pipeline/sync/test_phony
2022-09-27T15:46:44.9932090Z  distributed/pipeline/sync/test_stream
2022-09-27T15:46:44.9932384Z  distributed/rpc/test_faulty_agent
2022-09-27T15:46:45.0055639Z Prioritized test from test file changes.
2022-09-27T15:46:45.0056331Z reordering tests for PR:
2022-09-27T15:46:45.0056970Z prioritized: ['distributed/test_c10d_gloo', 'distributed/fsdp/test_fsdp_mixed_precision']
2022-09-27T15:46:45.0061604Z the rest: ['distributed/test_c10d_nccl', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/algorithms/quantization/test_quantization', 'distributed/test_pg_wrapper', 'distributed/test_store', 'distributed/algorithms/test_join', 'distributed/test_c10d_spawn_gloo', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/test_c10d_common', 'distributed/elastic/multiprocessing/api_test', 'distributed/_shard/checkpoint/test_checkpoint', 'distributed/fsdp/test_fsdp_param_exec_order_wrap', 'distributed/test_c10d_object_collectives', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/fsdp/test_fsdp_overlap', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/elastic/timer/local_timer_example', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_fx', 'distributed/test_data_parallel', 'distributed/rpc/test_share_memory', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/elastic/utils/util_test', 'distributed/nn/jit/test_instantiator', 'distributed/_shard/checkpoint/test_planner', 'distributed/fsdp/test_utils', 'distributed/elastic/metrics/api_test', 'distributed/_shard/test_replicated_tensor', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_stream', 'distributed/rpc/test_faulty_agent']
2022-09-27T15:46:45.0064646Z 
2022-09-27T15:46:45.0065196Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-09-27T15:46:45.0370251Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-09-27T15:46:45.0576330Z Running distributed/test_c10d_gloo ... [2022-09-27 15:46:45.057223]
2022-09-27T15:46:45.0577042Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:46:45.057288]
2022-09-27T15:46:46.9612136Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-09-27T15:46:46.9614361Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-09-27T15:46:46.9614730Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-09-27T15:46:46.9615371Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-09-27T15:46:46.9615769Z test_gloo_rank_membership (__main__.CommTest)
2022-09-27T15:46:46.9616031Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-09-27T15:46:46.9616444Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-09-27T15:46:46.9616858Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-09-27T15:46:46.9617236Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-09-27T15:46:46.9617595Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-09-27T15:46:46.9617953Z test_tensor_dtype_complex (__main__.CommTest)
2022-09-27T15:46:46.9618239Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-09-27T15:46:46.9619452Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-09-27T15:46:46.9620640Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:46:46.9621008Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:46:46.9621373Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:46:46.9621704Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:46:46.9622192Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:46:46.9622565Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:46:46.9622921Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:46:46.9623310Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:46:46.9623691Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-09-27T15:46:46.9624057Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2022-09-27T15:46:46.9624386Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-09-27T15:46:46.9629779Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-09-27T15:46:46.9635259Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9635765Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9636283Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9636779Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9637274Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9637809Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9638335Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9638831Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9639305Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9639818Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9640337Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9640840Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9641422Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9641914Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9642392Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9642860Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9643294Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9643760Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9644205Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9644743Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9645242Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9645727Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9646232Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9646712Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9647222Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9647698Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9648120Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9648560Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9649021Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9649461Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9650082Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9650544Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9650987Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9651391Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9651833Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9652282Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9652818Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T15:46:46.9653199Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:46.9658333Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-09-27T15:46:46.9663462Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9663862Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9664256Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9664656Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9665042Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9665516Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9665934Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9666305Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9666693Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9667087Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9667481Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9667921Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9668325Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9668720Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9669102Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9669513Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9669940Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9670339Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9670739Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9671130Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9671498Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9671886Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9672271Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9672655Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9673007Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9673392Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9673772Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9674116Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9674497Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9674873Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9675246Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9675706Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9676083Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9676456Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9676841Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9677225Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9677610Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9677953Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9678329Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9678704Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9679074Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9679460Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9679841Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9680198Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9680570Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9680950Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9681342Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9681719Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9682127Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9682538Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-09-27T15:46:46.9683506Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-09-27T15:46:46.9684315Z test_forward_backward (__main__.ReducerTest)
2022-09-27T15:46:46.9684672Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-09-27T15:46:46.9685059Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-09-27T15:46:46.9685413Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-09-27T15:46:46.9685776Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-09-27T15:46:46.9686141Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-09-27T15:46:46.9686575Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-09-27T15:46:46.9686971Z test_logging_init (__main__.RendezvousEnvTest)
2022-09-27T15:46:46.9687313Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:46.9687749Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-09-27T15:46:46.9688163Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-09-27T15:46:48.5926681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:48.5927226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:48.5928996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:48.5929517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:48.8524324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:46:48.8539057Z 
2022-09-27T15:46:48.8539313Z Running tests...
2022-09-27T15:46:48.8539751Z ----------------------------------------------------------------------
2022-09-27T15:46:50.4737790Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:46:50.4928303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 685
2022-09-27T15:46:50.4935705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 686
2022-09-27T15:46:52.1248017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:52.1248539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:52.1256916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:52.1257430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:52.1549676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:52.1550190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:52.1560885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:52.1561412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:52.3903381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:52.4115012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:52.8994782Z ok (4.045s)
2022-09-27T15:46:52.8995203Z 
2022-09-27T15:46:52.8995876Z ----------------------------------------------------------------------
2022-09-27T15:46:52.8996247Z Ran 1 test in 4.045s
2022-09-27T15:46:52.8996423Z 
2022-09-27T15:46:52.8996524Z OK
2022-09-27T15:46:52.8996667Z 
2022-09-27T15:46:52.8996781Z Generating XML reports...
2022-09-27T15:46:52.9035096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154648.xml
2022-09-27T15:46:54.9924631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:54.9925213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:54.9925819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:54.9926313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:55.2578460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:46:55.2593842Z 
2022-09-27T15:46:55.2593995Z Running tests...
2022-09-27T15:46:55.2594840Z ----------------------------------------------------------------------
2022-09-27T15:46:56.9389442Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:46:56.9587436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 794
2022-09-27T15:46:56.9595311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 795
2022-09-27T15:46:58.6490363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:58.6490944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:58.6498886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:58.6499379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:58.6727240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:58.6728208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:58.6739046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:58.6739957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:58.9238517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:58.9340193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:00.9688771Z ok (5.709s)
2022-09-27T15:47:00.9689255Z 
2022-09-27T15:47:00.9690154Z ----------------------------------------------------------------------
2022-09-27T15:47:00.9690569Z Ran 1 test in 5.709s
2022-09-27T15:47:00.9690742Z 
2022-09-27T15:47:00.9690819Z OK
2022-09-27T15:47:00.9690973Z 
2022-09-27T15:47:00.9691112Z Generating XML reports...
2022-09-27T15:47:00.9727846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154655.xml
2022-09-27T15:47:03.0684372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:03.0684898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:03.0686845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:03.0687354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:03.3310380Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:03.3325327Z 
2022-09-27T15:47:03.3325452Z Running tests...
2022-09-27T15:47:03.3326332Z ----------------------------------------------------------------------
2022-09-27T15:47:04.9474534Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:04.9666832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 905
2022-09-27T15:47:04.9673048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 906
2022-09-27T15:47:06.6865592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:06.6866429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:06.6873716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:06.6874231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:06.7055850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:06.7056307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:06.7066654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:06.7067135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:06.9638909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:06.9641853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:06.9852990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:06.9853563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:06.9854268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:06.9854979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:07.3731595Z ok (4.040s)
2022-09-27T15:47:07.3731871Z 
2022-09-27T15:47:07.3732276Z ----------------------------------------------------------------------
2022-09-27T15:47:07.3732625Z Ran 1 test in 4.040s
2022-09-27T15:47:07.3732783Z 
2022-09-27T15:47:07.3732878Z OK
2022-09-27T15:47:07.3733018Z 
2022-09-27T15:47:07.3733158Z Generating XML reports...
2022-09-27T15:47:07.3770606Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154703.xml
2022-09-27T15:47:09.4682376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:09.4683513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:09.4684129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:09.4684616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:09.7378916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:09.7394379Z 
2022-09-27T15:47:09.7394694Z Running tests...
2022-09-27T15:47:09.7395146Z ----------------------------------------------------------------------
2022-09-27T15:47:11.3832346Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:11.4029140Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1014
2022-09-27T15:47:11.4037045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1015
2022-09-27T15:47:13.0717637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:13.0718175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:13.0726824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:13.0727335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:13.0769162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:13.0769844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:13.0780395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:13.0780874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:13.3234760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:13.3492234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:13.3850396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:13.3850897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:13.3851674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:13.3852375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:13.3853574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:13.3959713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:13.3960414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:13.4058865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:13.8105913Z ok (4.071s)
2022-09-27T15:47:13.8106232Z 
2022-09-27T15:47:13.8106671Z ----------------------------------------------------------------------
2022-09-27T15:47:13.8107028Z Ran 1 test in 4.071s
2022-09-27T15:47:13.8107208Z 
2022-09-27T15:47:13.8107308Z OK
2022-09-27T15:47:13.8109477Z 
2022-09-27T15:47:13.8109925Z Generating XML reports...
2022-09-27T15:47:13.8146316Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154709.xml
2022-09-27T15:47:15.9288279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.9288815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.9290576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.9291073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:16.1910052Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:16.1924781Z 
2022-09-27T15:47:16.1925013Z Running tests...
2022-09-27T15:47:16.1925440Z ----------------------------------------------------------------------
2022-09-27T15:47:17.8034475Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:17.8225757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1126
2022-09-27T15:47:17.8233333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1127
2022-09-27T15:47:19.5246406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.5246965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.5255163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.5255658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.5273933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.5274394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.5285575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.5286073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.7877446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:19.7894886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:19.8214692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:19.8215184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:19.8216042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.8216745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.8217598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:19.8220984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:19.8222133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:19.8320549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:21.7327097Z ok (5.540s)
2022-09-27T15:47:21.7327566Z 
2022-09-27T15:47:21.7328275Z ----------------------------------------------------------------------
2022-09-27T15:47:21.7328621Z Ran 1 test in 5.540s
2022-09-27T15:47:21.7328798Z 
2022-09-27T15:47:21.7328899Z OK
2022-09-27T15:47:21.7329037Z 
2022-09-27T15:47:21.7329176Z Generating XML reports...
2022-09-27T15:47:21.7366514Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154716.xml
2022-09-27T15:47:23.8274333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:23.8274970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:23.8276417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:23.8277304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:24.0975736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:24.0991781Z 
2022-09-27T15:47:24.0992411Z Running tests...
2022-09-27T15:47:24.0993323Z ----------------------------------------------------------------------
2022-09-27T15:47:25.7553110Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:25.7752241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1240
2022-09-27T15:47:25.7760450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1241
2022-09-27T15:47:27.4240216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:27.4240804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:27.4248610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:27.4249365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:27.4520029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:27.4520607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:27.4533973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:27.4534686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:27.6973494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:27.7137190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:27.7498544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:27.7499715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:27.7500791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:27.7501504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:27.7709653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:27.7710730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:27.7711435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:27.7712140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:29.7853711Z ok (5.686s)
2022-09-27T15:47:29.7854165Z 
2022-09-27T15:47:29.7854803Z ----------------------------------------------------------------------
2022-09-27T15:47:29.7855193Z Ran 1 test in 5.686s
2022-09-27T15:47:29.7855366Z 
2022-09-27T15:47:29.7855469Z OK
2022-09-27T15:47:29.7855610Z 
2022-09-27T15:47:29.7855751Z Generating XML reports...
2022-09-27T15:47:29.7893437Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154724.xml
2022-09-27T15:47:31.8494759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:31.8495334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:31.8496819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:31.8497320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:32.1060056Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:32.1075428Z 
2022-09-27T15:47:32.1075803Z Running tests...
2022-09-27T15:47:32.1076536Z ----------------------------------------------------------------------
2022-09-27T15:47:33.7190840Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:33.7382772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1357
2022-09-27T15:47:33.7390567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1358
2022-09-27T15:47:35.3961248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:35.3962291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:35.3968980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:35.3970421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:35.4157588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:35.4158560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:35.4170440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:35.4171483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:35.6710278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:35.6821606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:36.1450663Z skip: Need at least 4 CUDA devices (4.037s)
2022-09-27T15:47:36.1450970Z 
2022-09-27T15:47:36.1451398Z ----------------------------------------------------------------------
2022-09-27T15:47:36.1451756Z Ran 1 test in 4.037s
2022-09-27T15:47:36.1451924Z 
2022-09-27T15:47:36.1452042Z OK (skipped=1)
2022-09-27T15:47:36.1452204Z 
2022-09-27T15:47:36.1452332Z Generating XML reports...
2022-09-27T15:47:36.1490461Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154732.xml
2022-09-27T15:47:38.2359337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:38.2359891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:38.2361157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:38.2361718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:38.4932787Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:38.4947940Z 
2022-09-27T15:47:38.4948034Z Running tests...
2022-09-27T15:47:38.4948892Z ----------------------------------------------------------------------
2022-09-27T15:47:40.1048836Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:40.1240536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1460
2022-09-27T15:47:40.1247284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1461
2022-09-27T15:47:41.7880334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:41.7880915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:41.7888819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:41.7889280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:41.8067565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:41.8068042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:41.8079151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:41.8079609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:42.0677105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:42.0756376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:42.1094873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:42.1095485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:42.1096306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:42.1097009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:42.5308505Z ok (4.036s)
2022-09-27T15:47:42.5308746Z 
2022-09-27T15:47:42.5309254Z ----------------------------------------------------------------------
2022-09-27T15:47:42.5309591Z Ran 1 test in 4.036s
2022-09-27T15:47:42.5309763Z 
2022-09-27T15:47:42.5309873Z OK
2022-09-27T15:47:42.5309926Z 
2022-09-27T15:47:42.5310067Z Generating XML reports...
2022-09-27T15:47:42.5347780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154738.xml
2022-09-27T15:47:44.6210909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:44.6211814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:44.6213159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:44.6213655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:44.8943407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:44.8958286Z 
2022-09-27T15:47:44.8958451Z Running tests...
2022-09-27T15:47:44.8959201Z ----------------------------------------------------------------------
2022-09-27T15:47:46.5561391Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:46.5760710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1569
2022-09-27T15:47:46.5768254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1570
2022-09-27T15:47:48.2197734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:48.2198336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:48.2206436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:48.2534232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:48.2534818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:48.2535244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:48.2545508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:48.2546015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:48.4923026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:48.5156821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:48.5473830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:48.5474521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:48.5475412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:48.5476157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:48.5682872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:47:48.5683664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:47:48.5684390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:48.5685173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:47:49.0831862Z ok (4.187s)
2022-09-27T15:47:49.0832142Z 
2022-09-27T15:47:49.0832575Z ----------------------------------------------------------------------
2022-09-27T15:47:49.0832940Z Ran 1 test in 4.187s
2022-09-27T15:47:49.0833149Z 
2022-09-27T15:47:49.0833282Z OK
2022-09-27T15:47:49.0833477Z 
2022-09-27T15:47:49.0833654Z Generating XML reports...
2022-09-27T15:47:49.0871909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154744.xml
2022-09-27T15:47:51.1998265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:51.1999624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:51.2002568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:51.2003118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:51.4763534Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:51.4779057Z 
2022-09-27T15:47:51.4779462Z Running tests...
2022-09-27T15:47:51.4779974Z ----------------------------------------------------------------------
2022-09-27T15:47:53.1490073Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:53.1690822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1684
2022-09-27T15:47:53.1757644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1685
2022-09-27T15:47:54.8674075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:54.8674626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:54.8682264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:54.8905338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:54.8905957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:54.8906439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:54.8916797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:54.8917291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:55.1469353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:55.1682628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:55.2087444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:55.2088314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:55.2089308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:55.2090291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:55.6761280Z ok (4.198s)
2022-09-27T15:47:55.6761528Z 
2022-09-27T15:47:55.6761941Z ----------------------------------------------------------------------
2022-09-27T15:47:55.6762273Z Ran 1 test in 4.198s
2022-09-27T15:47:55.6762453Z 
2022-09-27T15:47:55.6762555Z OK
2022-09-27T15:47:55.6762697Z 
2022-09-27T15:47:55.6762842Z Generating XML reports...
2022-09-27T15:47:55.6800804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154751.xml
2022-09-27T15:47:57.7994775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:57.7995343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:57.7996377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:57.7996883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:58.0646479Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:47:58.0662499Z 
2022-09-27T15:47:58.0662811Z Running tests...
2022-09-27T15:47:58.0663268Z ----------------------------------------------------------------------
2022-09-27T15:47:59.6816971Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:59.7008624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1793
2022-09-27T15:47:59.7016218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1794
2022-09-27T15:48:01.3466976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:01.3467563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:01.3477879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:01.3478378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:01.3781774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:01.3782278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:01.3793207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:01.3793702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:01.6240024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:01.6438932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:01.6754791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:01.6755754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:01.6756757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:01.6757481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:02.1076071Z ok (4.041s)
2022-09-27T15:48:02.1076321Z 
2022-09-27T15:48:02.1076798Z ----------------------------------------------------------------------
2022-09-27T15:48:02.1077099Z Ran 1 test in 4.041s
2022-09-27T15:48:02.1077618Z 
2022-09-27T15:48:02.1077723Z OK
2022-09-27T15:48:02.1077869Z 
2022-09-27T15:48:02.1078011Z Generating XML reports...
2022-09-27T15:48:02.1115962Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154758.xml
2022-09-27T15:48:04.2199330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:04.2199938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:04.2201345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:04.2201963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:04.4918797Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:04.4935559Z 
2022-09-27T15:48:04.4936041Z Running tests...
2022-09-27T15:48:04.4936563Z ----------------------------------------------------------------------
2022-09-27T15:48:06.1428155Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:06.1620543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1902
2022-09-27T15:48:06.1628589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1903
2022-09-27T15:48:07.8400449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:07.8401032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:07.8409437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:07.8410162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:07.8548826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:07.8549339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:07.8559793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:07.8560522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:08.1100713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:08.1181485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:08.1300557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:08.1301068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:08.1301863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:08.1302741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:08.1496219Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:08.1497370Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:08.1498358Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:08.1499225Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:08.5687789Z ok (4.075s)
2022-09-27T15:48:08.5688026Z 
2022-09-27T15:48:08.5688406Z ----------------------------------------------------------------------
2022-09-27T15:48:08.5689214Z Ran 1 test in 4.075s
2022-09-27T15:48:08.5689364Z 
2022-09-27T15:48:08.5689464Z OK
2022-09-27T15:48:08.5689978Z 
2022-09-27T15:48:08.5690222Z Generating XML reports...
2022-09-27T15:48:08.5729266Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154804.xml
2022-09-27T15:48:10.6601579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:10.6602091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:10.6603291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:10.6603815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:10.9261498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:10.9276705Z 
2022-09-27T15:48:10.9277114Z Running tests...
2022-09-27T15:48:10.9277593Z ----------------------------------------------------------------------
2022-09-27T15:48:12.5555362Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:12.5750907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2011
2022-09-27T15:48:12.5758433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2012
2022-09-27T15:48:14.2215380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:14.2215916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:14.2224169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:14.2224665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:14.2472656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:14.2473132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:14.2483743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:14.2484210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:14.4863618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:14.5047359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:15.9525274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:15.9525906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:15.9526735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:15.9527463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:15.9943329Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:15.9944123Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:15.9945020Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:15.9945729Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:16.5851101Z ok (5.657s)
2022-09-27T15:48:16.5851358Z 
2022-09-27T15:48:16.5851796Z ----------------------------------------------------------------------
2022-09-27T15:48:16.5852144Z Ran 1 test in 5.657s
2022-09-27T15:48:16.5852320Z 
2022-09-27T15:48:16.5852427Z OK
2022-09-27T15:48:16.5852574Z 
2022-09-27T15:48:16.5852715Z Generating XML reports...
2022-09-27T15:48:16.5891066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154810.xml
2022-09-27T15:48:18.6978964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:18.6979510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:18.6980590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:18.6981064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:18.9577963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:18.9592276Z 
2022-09-27T15:48:18.9592527Z Running tests...
2022-09-27T15:48:18.9592970Z ----------------------------------------------------------------------
2022-09-27T15:48:20.5808556Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:20.5999978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2122
2022-09-27T15:48:20.6006854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2123
2022-09-27T15:48:22.2348999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:22.2349529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:22.2357842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:22.2358361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:22.2625767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:22.2626241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:22.2637388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:22.2637891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:22.5054368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:22.5204614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:22.5378707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:22.5379251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:22.5380049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:22.5380760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:22.5651158Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:22.5652031Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:22.5652918Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:22.5653839Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:22.5654675Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:22.5655370Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:22.5656216Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:22.5656908Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:23.0066393Z ok (4.047s)
2022-09-27T15:48:23.0066582Z 
2022-09-27T15:48:23.0067002Z ----------------------------------------------------------------------
2022-09-27T15:48:23.0067449Z Ran 1 test in 4.047s
2022-09-27T15:48:23.0067556Z 
2022-09-27T15:48:23.0067660Z OK
2022-09-27T15:48:23.0067787Z 
2022-09-27T15:48:23.0067936Z Generating XML reports...
2022-09-27T15:48:23.0106896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154818.xml
2022-09-27T15:48:25.1197089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:25.1197618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:25.1198966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:25.1199547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:25.3896584Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:25.3911529Z 
2022-09-27T15:48:25.3911830Z Running tests...
2022-09-27T15:48:25.3912290Z ----------------------------------------------------------------------
2022-09-27T15:48:27.0572679Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:27.0772299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2231
2022-09-27T15:48:27.0779983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2232
2022-09-27T15:48:28.7315063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:28.7316020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:28.7325061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:28.7325943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:28.7430303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:28.7430816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:28.7441884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:28.7442407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:29.0011595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:29.0395143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:30.4604375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:30.4605317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:30.4606704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:30.4608293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:30.4806245Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:30.4807236Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:30.4808103Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:30.4808861Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:30.4809927Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:30.4811234Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:30.4812967Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:30.4814295Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:30.9872024Z ok (5.596s)
2022-09-27T15:48:30.9872235Z 
2022-09-27T15:48:30.9872947Z ----------------------------------------------------------------------
2022-09-27T15:48:30.9873313Z Ran 1 test in 5.596s
2022-09-27T15:48:30.9873499Z 
2022-09-27T15:48:30.9873613Z OK
2022-09-27T15:48:30.9873759Z 
2022-09-27T15:48:30.9873903Z Generating XML reports...
2022-09-27T15:48:30.9912289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154825.xml
2022-09-27T15:48:33.1001623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:33.1002170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:33.1004174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:33.1005007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:33.3621089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:33.3636653Z 
2022-09-27T15:48:33.3637073Z Running tests...
2022-09-27T15:48:33.3637584Z ----------------------------------------------------------------------
2022-09-27T15:48:34.9804629Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:34.9996935Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2342
2022-09-27T15:48:35.0004571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2343
2022-09-27T15:48:36.6305398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:36.6306414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:36.6315414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:36.6316379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:36.6654049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:36.6654961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:36.6665332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:36.6666302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:36.9000366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:36.9219193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:36.9424969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:36.9425888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:36.9427393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:36.9428790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:36.9593709Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:36.9594811Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:36.9596560Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:36.9598024Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:37.4065623Z ok (4.042s)
2022-09-27T15:48:37.4065897Z 
2022-09-27T15:48:37.4066273Z ----------------------------------------------------------------------
2022-09-27T15:48:37.4066639Z Ran 1 test in 4.043s
2022-09-27T15:48:37.4066814Z 
2022-09-27T15:48:37.4066915Z OK
2022-09-27T15:48:37.4067058Z 
2022-09-27T15:48:37.4067202Z Generating XML reports...
2022-09-27T15:48:37.4105128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154833.xml
2022-09-27T15:48:39.4838491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:39.4839084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:39.4840709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:39.4841216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:39.7706031Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:39.7720707Z 
2022-09-27T15:48:39.7720973Z Running tests...
2022-09-27T15:48:39.7721456Z ----------------------------------------------------------------------
2022-09-27T15:48:41.3950174Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:41.4141356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2451
2022-09-27T15:48:41.4149248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2452
2022-09-27T15:48:43.1226580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:43.1227100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:43.1235290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:43.1235829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:43.1251000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:43.1251751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:43.1262794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:43.1263300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:43.3887756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:43.3896351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:44.8546839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:44.8547408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:44.8548249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:44.8548952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:44.8742258Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:44.8742999Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:44.8743904Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:44.8745137Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:45.4243003Z ok (5.652s)
2022-09-27T15:48:45.4243247Z 
2022-09-27T15:48:45.4243698Z ----------------------------------------------------------------------
2022-09-27T15:48:45.4243998Z Ran 1 test in 5.652s
2022-09-27T15:48:45.4244173Z 
2022-09-27T15:48:45.4244277Z OK
2022-09-27T15:48:45.4244428Z 
2022-09-27T15:48:45.4244562Z Generating XML reports...
2022-09-27T15:48:45.4283191Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154839.xml
2022-09-27T15:48:47.5488184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:47.5488763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:47.5490472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:47.5491009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:47.8041132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:47.8055490Z 
2022-09-27T15:48:47.8055812Z Running tests...
2022-09-27T15:48:47.8056281Z ----------------------------------------------------------------------
2022-09-27T15:48:49.4193215Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:49.4384218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2562
2022-09-27T15:48:49.4391399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2563
2022-09-27T15:48:51.1478161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:51.1478708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:51.1486604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:51.1487092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:51.1629827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:51.1630320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:51.1641012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:51.1641486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:51.4226993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:51.4302511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:51.4449491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:51.4450218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:51.4451036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:51.4451744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:51.4732137Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4733067Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4734141Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4734910Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4735776Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4736506Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4737341Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4738073Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4738888Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4739637Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4740442Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4741148Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4741969Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4742708Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.4743665Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:51.4744400Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:51.9452629Z ok (4.139s)
2022-09-27T15:48:51.9453128Z 
2022-09-27T15:48:51.9453723Z ----------------------------------------------------------------------
2022-09-27T15:48:51.9454085Z Ran 1 test in 4.139s
2022-09-27T15:48:51.9454260Z 
2022-09-27T15:48:51.9454363Z OK
2022-09-27T15:48:51.9454507Z 
2022-09-27T15:48:51.9454651Z Generating XML reports...
2022-09-27T15:48:51.9491630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154847.xml
2022-09-27T15:48:54.0418849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:54.0419433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:54.0420566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:54.0421038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:54.3084026Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:48:54.3100969Z 
2022-09-27T15:48:54.3101290Z Running tests...
2022-09-27T15:48:54.3101724Z ----------------------------------------------------------------------
2022-09-27T15:48:55.9737110Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:55.9935715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2671
2022-09-27T15:48:55.9943278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2672
2022-09-27T15:48:57.6732641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:57.6733164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:57.6741671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:57.6742234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:57.6793651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:57.6794185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:57.6807179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:57.6807699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:57.9455483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:57.9484350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:59.4191426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:59.4191922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:59.4192798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:59.4193533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:59.4501111Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4502129Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4502991Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4503728Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4504588Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4505353Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4506191Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4506878Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4507734Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4508469Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4509396Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4510159Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4510990Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4511709Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:48:59.4512586Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:48:59.4513244Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:00.0047520Z ok (5.694s)
2022-09-27T15:49:00.0047756Z 
2022-09-27T15:49:00.0048214Z ----------------------------------------------------------------------
2022-09-27T15:49:00.0048556Z Ran 1 test in 5.695s
2022-09-27T15:49:00.0048724Z 
2022-09-27T15:49:00.0048838Z OK
2022-09-27T15:49:00.0048950Z 
2022-09-27T15:49:00.0049106Z Generating XML reports...
2022-09-27T15:49:00.0086730Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154854.xml
2022-09-27T15:49:02.1288393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:02.1288920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:02.1290482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:02.1291033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:02.3990940Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:02.4007230Z 
2022-09-27T15:49:02.4007350Z Running tests...
2022-09-27T15:49:02.4008200Z ----------------------------------------------------------------------
2022-09-27T15:49:04.0470996Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:04.0808020Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2782
2022-09-27T15:49:04.0815677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2783
2022-09-27T15:49:05.7469500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:05.7470506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:05.7477963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:05.7478972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:05.7588762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:05.7589742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:05.7600825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:05.7601809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:06.0092111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:06.0149547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:06.0315796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:06.0316796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:06.0318553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:06.0320009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:06.0485275Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:06.0486948Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:06.0488756Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:06.0490211Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:06.0491128Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:06.0491888Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:06.0493094Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:06.0494583Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:06.4875735Z ok (4.086s)
2022-09-27T15:49:06.4875989Z 
2022-09-27T15:49:06.4876462Z ----------------------------------------------------------------------
2022-09-27T15:49:06.4876827Z Ran 1 test in 4.087s
2022-09-27T15:49:06.4876974Z 
2022-09-27T15:49:06.4877086Z OK
2022-09-27T15:49:06.4877225Z 
2022-09-27T15:49:06.4877684Z Generating XML reports...
2022-09-27T15:49:06.4914611Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154902.xml
2022-09-27T15:49:08.6109894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:08.6110435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:08.6111728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:08.6112201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:08.8803350Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:08.8819606Z 
2022-09-27T15:49:08.8820088Z Running tests...
2022-09-27T15:49:08.8820680Z ----------------------------------------------------------------------
2022-09-27T15:49:10.5617114Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:10.5816574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2891
2022-09-27T15:49:10.5824118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2892
2022-09-27T15:49:12.2601499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:12.2602013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:12.2607489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:12.2608031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:12.2611828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:12.2612323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:12.2618862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:12.2619359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:12.5136572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:12.5397945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:12.5623660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:12.5624226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:12.5625047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:12.5625824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:12.5816983Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:12.5817855Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:12.5818731Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:12.5819469Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:12.9885142Z ok (4.106s)
2022-09-27T15:49:12.9885548Z 
2022-09-27T15:49:12.9886201Z ----------------------------------------------------------------------
2022-09-27T15:49:12.9886833Z Ran 1 test in 4.106s
2022-09-27T15:49:12.9887544Z 
2022-09-27T15:49:12.9887718Z OK
2022-09-27T15:49:12.9887978Z 
2022-09-27T15:49:12.9888221Z Generating XML reports...
2022-09-27T15:49:12.9926722Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154908.xml
2022-09-27T15:49:15.0591279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:15.0591784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:15.0593974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:15.0594515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:15.3150944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:15.3165521Z 
2022-09-27T15:49:15.3165713Z Running tests...
2022-09-27T15:49:15.3166381Z ----------------------------------------------------------------------
2022-09-27T15:49:16.9625928Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:16.9815477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3000
2022-09-27T15:49:16.9823170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3001
2022-09-27T15:49:18.5802990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.5803541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.5812317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.5813057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:18.6526044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.6526544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.6537793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.6538278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:18.8445603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:18.9068343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:20.3299525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:20.3300014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:20.3300853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:20.3301575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:20.3511122Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:20.3511903Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:20.3512782Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T15:49:20.3513517Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T15:49:20.8914954Z ok (5.575s)
2022-09-27T15:49:20.8915246Z 
2022-09-27T15:49:20.8915669Z ----------------------------------------------------------------------
2022-09-27T15:49:20.8916255Z Ran 1 test in 5.575s
2022-09-27T15:49:20.8916434Z 
2022-09-27T15:49:20.8916554Z OK
2022-09-27T15:49:20.8916647Z 
2022-09-27T15:49:20.8916794Z Generating XML reports...
2022-09-27T15:49:20.8954032Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154915.xml
2022-09-27T15:49:23.0092975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:23.0094744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:23.0095940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:23.0096965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:23.2676599Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:23.2692629Z 
2022-09-27T15:49:23.2692899Z Running tests...
2022-09-27T15:49:23.2693367Z ----------------------------------------------------------------------
2022-09-27T15:49:23.2699416Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T15:49:24.8874435Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:24.9067426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3111
2022-09-27T15:49:24.9075730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3112
2022-09-27T15:49:26.5515928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:26.5516828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:26.5525158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:26.5525674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:26.5711405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:26.5711898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:26.5721986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:26.5722488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:26.8281237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:26.8285378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:28.2689117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppds3c__2
2022-09-27T15:49:28.2690181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppds3c__2/_remote_module_non_scriptable.py
2022-09-27T15:49:28.2737955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lscnmq7
2022-09-27T15:49:28.2740921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lscnmq7/_remote_module_non_scriptable.py
2022-09-27T15:49:29.3180147Z ok (6.048s)
2022-09-27T15:49:29.3180378Z 
2022-09-27T15:49:29.3180787Z ----------------------------------------------------------------------
2022-09-27T15:49:29.3181151Z Ran 1 test in 6.049s
2022-09-27T15:49:29.3181305Z 
2022-09-27T15:49:29.3181408Z OK
2022-09-27T15:49:29.3181520Z 
2022-09-27T15:49:29.3181714Z Generating XML reports...
2022-09-27T15:49:29.3220495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154923.xml
2022-09-27T15:49:31.4243684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:31.4244197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:31.4245420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:31.4245921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:31.6827651Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:31.6843004Z 
2022-09-27T15:49:31.6843262Z Running tests...
2022-09-27T15:49:31.6843722Z ----------------------------------------------------------------------
2022-09-27T15:49:31.6850233Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:49:33.2911304Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:33.3104298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3226
2022-09-27T15:49:33.3111841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3227
2022-09-27T15:49:34.9982383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:34.9982941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:34.9990947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:34.9991449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:35.0338683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:35.0339185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:35.0350616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:35.0351128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:35.2707487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:35.2912465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:36.7620912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j2keyv2
2022-09-27T15:49:36.7621535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j2keyv2/_remote_module_non_scriptable.py
2022-09-27T15:49:36.7859072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg94vgvk
2022-09-27T15:49:36.7861790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg94vgvk/_remote_module_non_scriptable.py
2022-09-27T15:49:37.8213719Z ok (6.137s)
2022-09-27T15:49:37.8213978Z 
2022-09-27T15:49:37.8214423Z ----------------------------------------------------------------------
2022-09-27T15:49:37.8214734Z Ran 1 test in 6.137s
2022-09-27T15:49:37.8214908Z 
2022-09-27T15:49:37.8215008Z OK
2022-09-27T15:49:37.8215162Z 
2022-09-27T15:49:37.8215302Z Generating XML reports...
2022-09-27T15:49:37.8253767Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154931.xml
2022-09-27T15:49:39.9317819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:39.9318329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:39.9319581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:39.9320057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:40.2007634Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:40.2023582Z 
2022-09-27T15:49:40.2023824Z Running tests...
2022-09-27T15:49:40.2024248Z ----------------------------------------------------------------------
2022-09-27T15:49:40.2031852Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:49:41.8478643Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:41.8671755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3341
2022-09-27T15:49:41.8679521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3342
2022-09-27T15:49:43.5113925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.5114976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.5123680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.5124694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.5390817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.5391327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.5402128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.5402660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.7812315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:43.7966562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:45.2565747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndfk55yv
2022-09-27T15:49:45.2567222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndfk55yv/_remote_module_non_scriptable.py
2022-09-27T15:49:45.2833443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4ny_tsp
2022-09-27T15:49:45.2836115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4ny_tsp/_remote_module_non_scriptable.py
2022-09-27T15:49:45.7668095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.7668695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.7994798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.7995808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8160215Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:49:45.8161684Z   warnings.warn(
2022-09-27T15:49:45.8163683Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:49:45.8164464Z   warnings.warn(
2022-09-27T15:49:45.8279335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8280338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8508663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8509658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8828441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.8829876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.9104398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.9105383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:46.4796166Z ok (6.277s)
2022-09-27T15:49:46.4796442Z 
2022-09-27T15:49:46.4796891Z ----------------------------------------------------------------------
2022-09-27T15:49:46.4797169Z Ran 1 test in 6.277s
2022-09-27T15:49:46.4797346Z 
2022-09-27T15:49:46.4797446Z OK
2022-09-27T15:49:46.4797590Z 
2022-09-27T15:49:46.4797750Z Generating XML reports...
2022-09-27T15:49:46.4836695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154940.xml
2022-09-27T15:49:48.6180106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:48.6180689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:48.6182018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:48.6182434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:48.8750950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:48.8765649Z 
2022-09-27T15:49:48.8765892Z Running tests...
2022-09-27T15:49:48.8766350Z ----------------------------------------------------------------------
2022-09-27T15:49:48.8773729Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:49:50.4865037Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:50.5056915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3456
2022-09-27T15:49:50.5064237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3457
2022-09-27T15:49:52.1418767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.1419707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.1428640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.1429490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.1785973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.1786459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.1797589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.1798101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.4057225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:52.4355586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:53.8746784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkspxk5la
2022-09-27T15:49:53.8747837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkspxk5la/_remote_module_non_scriptable.py
2022-09-27T15:49:53.9105639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxr9ppgq
2022-09-27T15:49:53.9108344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxr9ppgq/_remote_module_non_scriptable.py
2022-09-27T15:49:54.3911922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.3912652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4243465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4244001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4411272Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:49:54.4412004Z   warnings.warn(
2022-09-27T15:49:54.4413050Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:49:54.4413796Z   warnings.warn(
2022-09-27T15:49:54.4529515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4530302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4757872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.4758405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.5082880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.5083588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.5362122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:54.5362658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:55.1170068Z ok (6.240s)
2022-09-27T15:49:55.1170332Z 
2022-09-27T15:49:55.1170726Z ----------------------------------------------------------------------
2022-09-27T15:49:55.1171067Z Ran 1 test in 6.240s
2022-09-27T15:49:55.1171217Z 
2022-09-27T15:49:55.1171319Z OK
2022-09-27T15:49:55.1171493Z 
2022-09-27T15:49:55.1171634Z Generating XML reports...
2022-09-27T15:49:55.1209906Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154948.xml
2022-09-27T15:49:57.1797869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:57.1798439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:57.1799735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:57.1800240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:57.4503328Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:49:57.4519217Z 
2022-09-27T15:49:57.4519572Z Running tests...
2022-09-27T15:49:57.4520012Z ----------------------------------------------------------------------
2022-09-27T15:49:57.4525782Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:49:59.0978455Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:59.1169439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3571
2022-09-27T15:49:59.1177350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3572
2022-09-27T15:50:00.8006854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.8007765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.8015562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:00.8016033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:00.8422897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.8423865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.8434288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:00.8435309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:01.0661012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:01.0977025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:02.5586100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf02kqvda
2022-09-27T15:50:02.5587264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf02kqvda/_remote_module_non_scriptable.py
2022-09-27T15:50:02.6257339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyl7km843
2022-09-27T15:50:02.6258367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyl7km843/_remote_module_non_scriptable.py
2022-09-27T15:50:03.0954559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:03.0955128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:03.1287534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:03.1288087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:03.7279380Z ok (6.276s)
2022-09-27T15:50:03.7279647Z 
2022-09-27T15:50:03.7280091Z ----------------------------------------------------------------------
2022-09-27T15:50:03.7280421Z Ran 1 test in 6.276s
2022-09-27T15:50:03.7280596Z 
2022-09-27T15:50:03.7280700Z OK
2022-09-27T15:50:03.7280840Z 
2022-09-27T15:50:03.7280985Z Generating XML reports...
2022-09-27T15:50:03.7318888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154957.xml
2022-09-27T15:50:05.8442332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:05.8443272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:05.8444716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:05.8445516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:06.1041847Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:06.1057451Z 
2022-09-27T15:50:06.1057735Z Running tests...
2022-09-27T15:50:06.1058161Z ----------------------------------------------------------------------
2022-09-27T15:50:06.1064470Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:50:07.7185655Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:07.7377221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3686
2022-09-27T15:50:07.7385485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3687
2022-09-27T15:50:09.4078322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:09.4079409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:09.4087157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:09.4088578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:09.4361663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:09.4362651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:09.4373939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:09.4374916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:09.6776798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:09.6851245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:11.1341121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tqo16tp
2022-09-27T15:50:11.1342384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tqo16tp/_remote_module_non_scriptable.py
2022-09-27T15:50:11.1374778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi2iimm1
2022-09-27T15:50:11.1377761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi2iimm1/_remote_module_non_scriptable.py
2022-09-27T15:50:11.6013666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:11.6014735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:11.6351672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:11.6353154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:12.2487575Z ok (6.143s)
2022-09-27T15:50:12.2487855Z 
2022-09-27T15:50:12.2488318Z ----------------------------------------------------------------------
2022-09-27T15:50:12.2488701Z Ran 1 test in 6.143s
2022-09-27T15:50:12.2488874Z 
2022-09-27T15:50:12.2488975Z OK
2022-09-27T15:50:12.2489115Z 
2022-09-27T15:50:12.2489233Z Generating XML reports...
2022-09-27T15:50:12.2534835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155006.xml
2022-09-27T15:50:14.3060471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:14.3061000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:14.3063026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:14.3063522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:14.5617921Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:14.5633897Z 
2022-09-27T15:50:14.5634060Z Running tests...
2022-09-27T15:50:14.5634790Z ----------------------------------------------------------------------
2022-09-27T15:50:14.5643220Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:50:16.1789888Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:16.1980211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3801
2022-09-27T15:50:16.1987922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3802
2022-09-27T15:50:17.8762415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:17.8762939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:17.8770707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:17.8771523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:17.8967765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:17.8968237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:17.8979796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:17.8980260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:18.1497963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:18.1578962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:19.6222594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lrl9l18
2022-09-27T15:50:19.6223446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lrl9l18/_remote_module_non_scriptable.py
2022-09-27T15:50:19.6350974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw8r_5uj
2022-09-27T15:50:19.6354137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw8r_5uj/_remote_module_non_scriptable.py
2022-09-27T15:50:20.1019965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:20.1020545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:20.1289324Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:50:20.1291339Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:50:20.1689054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:20.1689773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:20.8096205Z ok (6.246s)
2022-09-27T15:50:20.8096451Z 
2022-09-27T15:50:20.8096883Z ----------------------------------------------------------------------
2022-09-27T15:50:20.8097209Z Ran 1 test in 6.246s
2022-09-27T15:50:20.8097382Z 
2022-09-27T15:50:20.8097458Z OK
2022-09-27T15:50:20.8097597Z 
2022-09-27T15:50:20.8097740Z Generating XML reports...
2022-09-27T15:50:20.8136115Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155014.xml
2022-09-27T15:50:22.9261648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:22.9262175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:22.9263231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:22.9263722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.1838117Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:23.1854179Z 
2022-09-27T15:50:23.1854491Z Running tests...
2022-09-27T15:50:23.1855154Z ----------------------------------------------------------------------
2022-09-27T15:50:23.1863033Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:50:24.8053982Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:24.8244476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3916
2022-09-27T15:50:24.8252847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3917
2022-09-27T15:50:26.4968691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:26.4969230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:26.4977923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:26.4978406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:26.4995487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:26.4995946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:26.5007390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:26.5007836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:26.7607638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:26.7654506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:28.2559172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15otco3w
2022-09-27T15:50:28.2560331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15otco3w/_remote_module_non_scriptable.py
2022-09-27T15:50:28.2635466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a58vwdt
2022-09-27T15:50:28.2638387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a58vwdt/_remote_module_non_scriptable.py
2022-09-27T15:50:29.3358896Z ok (6.150s)
2022-09-27T15:50:29.3359151Z 
2022-09-27T15:50:29.3359590Z ----------------------------------------------------------------------
2022-09-27T15:50:29.3359947Z Ran 1 test in 6.150s
2022-09-27T15:50:29.3360119Z 
2022-09-27T15:50:29.3360197Z OK
2022-09-27T15:50:29.3360344Z 
2022-09-27T15:50:29.3360493Z Generating XML reports...
2022-09-27T15:50:29.3398695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155023.xml
2022-09-27T15:50:31.4116952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:31.4117492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:31.4119244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:31.4119715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.6732838Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:31.6748180Z 
2022-09-27T15:50:31.6748602Z Running tests...
2022-09-27T15:50:31.6749082Z ----------------------------------------------------------------------
2022-09-27T15:50:31.6754492Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T15:50:33.2767102Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:33.2959570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4031
2022-09-27T15:50:33.2966562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4032
2022-09-27T15:50:34.9566894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.9567620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.9575259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.9575754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:34.9971555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.9972024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.9984005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.9984683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:35.2175268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:35.2541515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:36.6963746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0m00bfe
2022-09-27T15:50:36.6964485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0m00bfe/_remote_module_non_scriptable.py
2022-09-27T15:50:36.7390274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzynox024
2022-09-27T15:50:36.7392726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzynox024/_remote_module_non_scriptable.py
2022-09-27T15:50:37.2164588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:37.2165436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:37.2480379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:37.2480884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:37.8069162Z ok (6.132s)
2022-09-27T15:50:37.8069459Z 
2022-09-27T15:50:37.8070115Z ----------------------------------------------------------------------
2022-09-27T15:50:37.8070622Z Ran 1 test in 6.132s
2022-09-27T15:50:37.8070797Z 
2022-09-27T15:50:37.8070902Z OK
2022-09-27T15:50:37.8071018Z 
2022-09-27T15:50:37.8071160Z Generating XML reports...
2022-09-27T15:50:37.8108660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155031.xml
2022-09-27T15:50:39.8931214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:39.8931748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:39.8934166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:39.8934652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:40.1515862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:40.1530672Z 
2022-09-27T15:50:40.1531168Z Running tests...
2022-09-27T15:50:40.1531645Z ----------------------------------------------------------------------
2022-09-27T15:50:40.1539744Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:50:41.7781546Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:41.7971684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4146
2022-09-27T15:50:41.7979802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4147
2022-09-27T15:50:43.4648314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:43.4657450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:43.4658074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:43.4658521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:43.5045642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:43.5046121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:43.5057247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:43.5057743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:43.7338650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:43.7591431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:45.2268069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8qs5cbm
2022-09-27T15:50:45.2268969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8qs5cbm/_remote_module_non_scriptable.py
2022-09-27T15:50:45.2393647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cn7f5cg
2022-09-27T15:50:45.2396626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cn7f5cg/_remote_module_non_scriptable.py
2022-09-27T15:50:45.6882330Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:50:45.7007296Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:50:45.7299976Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:50:45.7300742Z   warnings.warn(
2022-09-27T15:50:45.7301806Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:50:45.7302528Z   warnings.warn(
2022-09-27T15:50:45.7416893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:45.7417376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:45.7969403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:45.7970119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:46.4084401Z ok (6.255s)
2022-09-27T15:50:46.4084843Z 
2022-09-27T15:50:46.4085340Z ----------------------------------------------------------------------
2022-09-27T15:50:46.4085651Z Ran 1 test in 6.255s
2022-09-27T15:50:46.4085817Z 
2022-09-27T15:50:46.4085916Z OK
2022-09-27T15:50:46.4086054Z 
2022-09-27T15:50:46.4086196Z Generating XML reports...
2022-09-27T15:50:46.4124201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155040.xml
2022-09-27T15:50:48.4900553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.4901126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.4902172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.4902668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.7504120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:48.7518803Z 
2022-09-27T15:50:48.7519292Z Running tests...
2022-09-27T15:50:48.7519818Z ----------------------------------------------------------------------
2022-09-27T15:50:48.7527467Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:50:50.3722518Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:50.3914482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4261
2022-09-27T15:50:50.3922227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4262
2022-09-27T15:50:52.0601981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.0602519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.0610880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.0611378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.0765229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.0765710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.0775913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.0776399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.3318890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:52.3319440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:53.7694898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45o_bdb0
2022-09-27T15:50:53.7695917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45o_bdb0/_remote_module_non_scriptable.py
2022-09-27T15:50:53.7949862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvzkg2yl
2022-09-27T15:50:53.7952507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvzkg2yl/_remote_module_non_scriptable.py
2022-09-27T15:50:54.2583774Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:50:54.2584572Z   warnings.warn(
2022-09-27T15:50:54.2585645Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:50:54.2586688Z   warnings.warn(
2022-09-27T15:50:54.2716286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:54.2716785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:54.3131224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:54.3131730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:54.9024620Z ok (6.150s)
2022-09-27T15:50:54.9024889Z 
2022-09-27T15:50:54.9025345Z ----------------------------------------------------------------------
2022-09-27T15:50:54.9025686Z Ran 1 test in 6.150s
2022-09-27T15:50:54.9025878Z 
2022-09-27T15:50:54.9025979Z OK
2022-09-27T15:50:54.9026117Z 
2022-09-27T15:50:54.9026260Z Generating XML reports...
2022-09-27T15:50:54.9064678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155048.xml
2022-09-27T15:50:56.9665052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:56.9665628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:56.9666444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:56.9666903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:57.2231751Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:50:57.2246930Z 
2022-09-27T15:50:57.2247402Z Running tests...
2022-09-27T15:50:57.2247935Z ----------------------------------------------------------------------
2022-09-27T15:50:57.2258128Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T15:50:58.8468451Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:58.8659166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4376
2022-09-27T15:50:58.8667271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4377
2022-09-27T15:51:00.5052460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:00.5053479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:00.5062681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:00.5063626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:00.5448035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:00.5448945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:00.5460385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:00.5461341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:00.7681465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:00.8029758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:02.2661546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjh4qj3o
2022-09-27T15:51:02.2662673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjh4qj3o/_remote_module_non_scriptable.py
2022-09-27T15:51:02.2754252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jkpv5xy
2022-09-27T15:51:02.2757528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jkpv5xy/_remote_module_non_scriptable.py
2022-09-27T15:51:02.7417338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:02.7417883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:02.7796210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:02.7797172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:03.3772002Z ok (6.152s)
2022-09-27T15:51:03.3772226Z 
2022-09-27T15:51:03.3772651Z ----------------------------------------------------------------------
2022-09-27T15:51:03.3773025Z Ran 1 test in 6.152s
2022-09-27T15:51:03.3773152Z 
2022-09-27T15:51:03.3773252Z OK
2022-09-27T15:51:03.3773391Z 
2022-09-27T15:51:03.3773546Z Generating XML reports...
2022-09-27T15:51:03.3810965Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155057.xml
2022-09-27T15:51:05.5118104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:05.5118632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:05.5119257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:05.5119713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:05.7804088Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:05.7820199Z 
2022-09-27T15:51:05.7820664Z Running tests...
2022-09-27T15:51:05.7821084Z ----------------------------------------------------------------------
2022-09-27T15:51:05.7830896Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T15:51:07.4576372Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:07.4776439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4491
2022-09-27T15:51:07.4784547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4492
2022-09-27T15:51:09.1107884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:09.1108428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:09.1116779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:09.1117291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:09.1407476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:09.1407994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:09.1420003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:09.1420512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:09.3805733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:09.3966264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:10.8532158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho4wasgr
2022-09-27T15:51:10.8533298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho4wasgr/_remote_module_non_scriptable.py
2022-09-27T15:51:10.8999801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsrgi5sh
2022-09-27T15:51:10.9001911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsrgi5sh/_remote_module_non_scriptable.py
2022-09-27T15:51:11.3679190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.3679819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.3999263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.3999744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.4220260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.4220756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.4535226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:11.4535702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:51:12.0905967Z ok (6.308s)
2022-09-27T15:51:12.0906244Z 
2022-09-27T15:51:12.0906635Z ----------------------------------------------------------------------
2022-09-27T15:51:12.0907009Z Ran 1 test in 6.308s
2022-09-27T15:51:12.0907191Z 
2022-09-27T15:51:12.0907301Z OK
2022-09-27T15:51:12.0907430Z 
2022-09-27T15:51:12.0907571Z Generating XML reports...
2022-09-27T15:51:12.0946013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155105.xml
2022-09-27T15:51:14.1564554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.1565144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.1566258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.1566773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.4259417Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:14.4274129Z 
2022-09-27T15:51:14.4274597Z Running tests...
2022-09-27T15:51:14.4275122Z ----------------------------------------------------------------------
2022-09-27T15:51:14.4280949Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-09-27T15:51:16.0535356Z This unit test verifies whether the Future object is passed properly. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:16.0725147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4606
2022-09-27T15:51:16.0732382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4607
2022-09-27T15:51:17.7375160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.7375732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.7384137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.7384627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.7675235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.7675935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.7686866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.7687358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:18.0084293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:18.0284528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:18.0501844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt5r0ben
2022-09-27T15:51:18.0504508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt5r0ben/_remote_module_non_scriptable.py
2022-09-27T15:51:18.0505055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g1vbhqc
2022-09-27T15:51:18.0508055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g1vbhqc/_remote_module_non_scriptable.py
2022-09-27T15:51:18.5796026Z ok (4.152s)
2022-09-27T15:51:18.5796328Z 
2022-09-27T15:51:18.5796745Z ----------------------------------------------------------------------
2022-09-27T15:51:18.5797032Z Ran 1 test in 4.152s
2022-09-27T15:51:18.5797203Z 
2022-09-27T15:51:18.5797302Z OK
2022-09-27T15:51:18.5797418Z 
2022-09-27T15:51:18.5797561Z Generating XML reports...
2022-09-27T15:51:18.5836029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155114.xml
2022-09-27T15:51:20.7087821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:20.7088379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:20.7090101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:20.7090862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:20.9684881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:20.9700153Z 
2022-09-27T15:51:20.9700358Z Running tests...
2022-09-27T15:51:20.9701156Z ----------------------------------------------------------------------
2022-09-27T15:51:20.9706547Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-09-27T15:51:22.5896606Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:22.6087841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4719
2022-09-27T15:51:22.6095486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4720
2022-09-27T15:51:24.2414391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:24.2414925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:24.2423277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:24.2423777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:24.2741887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:24.2742373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:24.2753561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:24.2754024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:24.5053881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:24.5312405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:26.0021145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiduv89te
2022-09-27T15:51:26.0021815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiduv89te/_remote_module_non_scriptable.py
2022-09-27T15:51:26.0135950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh3b1a98
2022-09-27T15:51:26.0139046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh3b1a98/_remote_module_non_scriptable.py
2022-09-27T15:51:26.5186474Z ok (5.548s)
2022-09-27T15:51:26.5186959Z 
2022-09-27T15:51:26.5187626Z ----------------------------------------------------------------------
2022-09-27T15:51:26.5188325Z Ran 1 test in 5.548s
2022-09-27T15:51:26.5188511Z 
2022-09-27T15:51:26.5188682Z OK
2022-09-27T15:51:26.5188945Z 
2022-09-27T15:51:26.5189459Z Generating XML reports...
2022-09-27T15:51:26.5229056Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155120.xml
2022-09-27T15:51:28.6307098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:28.6307607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:28.6309552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:28.6310055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:28.9031730Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:28.9047412Z 
2022-09-27T15:51:28.9047705Z Running tests...
2022-09-27T15:51:28.9048164Z ----------------------------------------------------------------------
2022-09-27T15:51:28.9055679Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-09-27T15:51:30.5646180Z DDP communication hook can only be registered once. This test validates whether ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:30.5847172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4834
2022-09-27T15:51:30.5855131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4835
2022-09-27T15:51:32.1726596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:32.1727638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:32.1737538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:32.1738565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:32.2698200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:32.2699294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:32.2709817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:32.2710805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:32.4155290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:32.5203713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:32.5478924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ylb_sba
2022-09-27T15:51:32.5479869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzsdwacw
2022-09-27T15:51:32.5480977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ylb_sba/_remote_module_non_scriptable.py
2022-09-27T15:51:32.5482607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzsdwacw/_remote_module_non_scriptable.py
2022-09-27T15:51:32.9921259Z ok (4.087s)
2022-09-27T15:51:32.9921538Z 
2022-09-27T15:51:32.9921938Z ----------------------------------------------------------------------
2022-09-27T15:51:32.9922301Z Ran 1 test in 4.087s
2022-09-27T15:51:32.9922465Z 
2022-09-27T15:51:32.9922576Z OK
2022-09-27T15:51:32.9922718Z 
2022-09-27T15:51:32.9922826Z Generating XML reports...
2022-09-27T15:51:32.9960200Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155128.xml
2022-09-27T15:51:35.0917880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:35.0918403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:35.0920616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:35.0921119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:35.3623914Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:35.3639188Z 
2022-09-27T15:51:35.3639682Z Running tests...
2022-09-27T15:51:35.3640182Z ----------------------------------------------------------------------
2022-09-27T15:51:35.3649269Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-09-27T15:51:37.0179149Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:37.0377481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4943
2022-09-27T15:51:37.0385758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4944
2022-09-27T15:51:38.6623742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:38.6624269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:38.6632337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:38.6632848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:38.6989380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:38.6989869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:38.7001305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:38.7001869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:38.9327838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:38.9550728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:38.9856069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rwbe44d
2022-09-27T15:51:38.9858994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rwbe44d/_remote_module_non_scriptable.py
2022-09-27T15:51:38.9859691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr05i2mi
2022-09-27T15:51:38.9862056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr05i2mi/_remote_module_non_scriptable.py
2022-09-27T15:51:39.4444689Z ok (4.080s)
2022-09-27T15:51:39.4444947Z 
2022-09-27T15:51:39.4445411Z ----------------------------------------------------------------------
2022-09-27T15:51:39.4445769Z Ran 1 test in 4.080s
2022-09-27T15:51:39.4445918Z 
2022-09-27T15:51:39.4446015Z OK
2022-09-27T15:51:39.4446150Z 
2022-09-27T15:51:39.4446313Z Generating XML reports...
2022-09-27T15:51:39.4485231Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155135.xml
2022-09-27T15:51:41.5613347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:41.5613910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:41.5615327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:41.5615815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:41.8195728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:41.8211195Z 
2022-09-27T15:51:41.8211620Z Running tests...
2022-09-27T15:51:41.8212080Z ----------------------------------------------------------------------
2022-09-27T15:51:41.8220311Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-09-27T15:51:43.4170834Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:43.4362357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5086
2022-09-27T15:51:43.4369990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5087
2022-09-27T15:51:45.0727775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:45.0728312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:45.0736887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:45.0737394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:45.1023817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:45.1024308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:45.1034805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:45.1035262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:45.3506276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:45.3609334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:45.3927512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg388rjuv
2022-09-27T15:51:45.3928050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz7b5tf4
2022-09-27T15:51:45.3930606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg388rjuv/_remote_module_non_scriptable.py
2022-09-27T15:51:45.3931231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz7b5tf4/_remote_module_non_scriptable.py
2022-09-27T15:51:45.8427914Z ok (4.021s)
2022-09-27T15:51:45.8428208Z 
2022-09-27T15:51:45.8428618Z ----------------------------------------------------------------------
2022-09-27T15:51:45.8429128Z Ran 1 test in 4.022s
2022-09-27T15:51:45.8429396Z 
2022-09-27T15:51:45.8429503Z OK
2022-09-27T15:51:45.8429647Z 
2022-09-27T15:51:45.8429769Z Generating XML reports...
2022-09-27T15:51:45.8467761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155141.xml
2022-09-27T15:51:47.9539183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:47.9539727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:47.9541119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:47.9541622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:48.2204294Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:48.2220544Z 
2022-09-27T15:51:48.2221049Z Running tests...
2022-09-27T15:51:48.2221575Z ----------------------------------------------------------------------
2022-09-27T15:51:48.2232362Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-09-27T15:51:49.8742324Z This test checks whether return annotation checked properly if defined. It also ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:49.8941763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5195
2022-09-27T15:51:49.8950256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5196
2022-09-27T15:51:51.5946458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:51.5947325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:51.5954576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:51.5955073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:51.6013587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:51.6014038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:51.6024958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:51.6025449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:51.8608595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:51.8617285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:51.8931083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquso788u
2022-09-27T15:51:51.8933867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquso788u/_remote_module_non_scriptable.py
2022-09-27T15:51:51.8935739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu5144fj
2022-09-27T15:51:51.8938593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu5144fj/_remote_module_non_scriptable.py
2022-09-27T15:51:52.4013171Z ok (4.179s)
2022-09-27T15:51:52.4013380Z 
2022-09-27T15:51:52.4013818Z ----------------------------------------------------------------------
2022-09-27T15:51:52.4014169Z Ran 1 test in 4.179s
2022-09-27T15:51:52.4014324Z 
2022-09-27T15:51:52.4014443Z OK
2022-09-27T15:51:52.4014563Z 
2022-09-27T15:51:52.4015015Z Generating XML reports...
2022-09-27T15:51:52.4055598Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155148.xml
2022-09-27T15:51:54.5060105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:54.5060648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:54.5062771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:54.5063275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:54.7730410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:51:54.7746857Z 
2022-09-27T15:51:54.7747304Z Running tests...
2022-09-27T15:51:54.7747812Z ----------------------------------------------------------------------
2022-09-27T15:51:54.7762289Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-09-27T15:51:56.4389510Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:56.4587836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5308
2022-09-27T15:51:56.4595834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5309
2022-09-27T15:51:58.1283639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:58.1284187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:58.1292286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:58.1292788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:58.1421643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:58.1422121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:58.1433269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:58.1433760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:58.3997064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:58.4077664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:58.4610756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfgnzjab
2022-09-27T15:51:58.4611949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1bbuly5
2022-09-27T15:51:58.4613553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfgnzjab/_remote_module_non_scriptable.py
2022-09-27T15:51:58.4615487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1bbuly5/_remote_module_non_scriptable.py
2022-09-27T15:51:58.4778441Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T15:52:00.4691215Z ok (5.694s)
2022-09-27T15:52:00.4691653Z 
2022-09-27T15:52:00.4692464Z ----------------------------------------------------------------------
2022-09-27T15:52:00.4692844Z Ran 1 test in 5.694s
2022-09-27T15:52:00.4693015Z 
2022-09-27T15:52:00.4693115Z OK
2022-09-27T15:52:00.4693258Z 
2022-09-27T15:52:00.4693722Z Generating XML reports...
2022-09-27T15:52:00.4729917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155154.xml
2022-09-27T15:52:02.5489535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:02.5490917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:02.5492690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:02.5493584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:02.8067755Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:02.8082437Z 
2022-09-27T15:52:02.8082874Z Running tests...
2022-09-27T15:52:02.8083367Z ----------------------------------------------------------------------
2022-09-27T15:52:04.4146248Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:04.4336747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5423
2022-09-27T15:52:04.4344226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5424
2022-09-27T15:52:06.0785590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:06.0786142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:06.0797819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:06.0798348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:06.1074218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:06.1074707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:06.1085687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:06.1086504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:06.3502036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:06.3663473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:06.4196170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnkxw0g6
2022-09-27T15:52:06.4196772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8b9mlk6l
2022-09-27T15:52:06.4198695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnkxw0g6/_remote_module_non_scriptable.py
2022-09-27T15:52:06.4199539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8b9mlk6l/_remote_module_non_scriptable.py
2022-09-27T15:52:08.4436569Z ok (5.635s)
2022-09-27T15:52:08.4436860Z 
2022-09-27T15:52:08.4437297Z ----------------------------------------------------------------------
2022-09-27T15:52:08.4437692Z Ran 1 test in 5.635s
2022-09-27T15:52:08.4437868Z 
2022-09-27T15:52:08.4437970Z OK
2022-09-27T15:52:08.4438111Z 
2022-09-27T15:52:08.4438233Z Generating XML reports...
2022-09-27T15:52:08.4476192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155202.xml
2022-09-27T15:52:10.5357210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:10.5357927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:10.5358557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:10.5359317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:10.7945213Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:10.7958969Z 
2022-09-27T15:52:10.7959689Z Running tests...
2022-09-27T15:52:10.7960325Z ----------------------------------------------------------------------
2022-09-27T15:52:12.4138530Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:12.4329743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5538
2022-09-27T15:52:12.4337249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5539
2022-09-27T15:52:14.1225130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:14.1225668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:14.1233985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:14.1234548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:14.1310119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:14.1310623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:14.1321514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:14.1321993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:14.3943483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:14.3989344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:14.4521232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0q7ljh7k
2022-09-27T15:52:14.4523541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0q7ljh7k/_remote_module_non_scriptable.py
2022-09-27T15:52:14.4524119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlp3bje2
2022-09-27T15:52:14.4526392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlp3bje2/_remote_module_non_scriptable.py
2022-09-27T15:52:16.4430471Z ok (5.647s)
2022-09-27T15:52:16.4430717Z 
2022-09-27T15:52:16.4431184Z ----------------------------------------------------------------------
2022-09-27T15:52:16.4431566Z Ran 1 test in 5.647s
2022-09-27T15:52:16.4431681Z 
2022-09-27T15:52:16.4431785Z OK
2022-09-27T15:52:16.4431936Z 
2022-09-27T15:52:16.4432082Z Generating XML reports...
2022-09-27T15:52:16.4469353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155210.xml
2022-09-27T15:52:18.5430386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:18.5430968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:18.5431998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:18.5432478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:18.8116299Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:18.8131482Z 
2022-09-27T15:52:18.8131938Z Running tests...
2022-09-27T15:52:18.8132463Z ----------------------------------------------------------------------
2022-09-27T15:52:20.4635585Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:20.4834032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5653
2022-09-27T15:52:20.4841564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5654
2022-09-27T15:52:22.1187632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:22.1188248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:22.1196585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:22.1197071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.1583587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:22.1584055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:22.1595200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:22.1595683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.3850761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:22.4172374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:22.4680995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudjb94bc
2022-09-27T15:52:22.4683053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_1o3vir
2022-09-27T15:52:22.4683614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudjb94bc/_remote_module_non_scriptable.py
2022-09-27T15:52:22.4685813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_1o3vir/_remote_module_non_scriptable.py
2022-09-27T15:52:22.4850688Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:52:22.4851421Z   warnings.warn(
2022-09-27T15:52:22.4852490Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T15:52:22.4853468Z   warnings.warn(
2022-09-27T15:52:24.4931931Z ok (5.680s)
2022-09-27T15:52:24.4932217Z 
2022-09-27T15:52:24.4932644Z ----------------------------------------------------------------------
2022-09-27T15:52:24.4932973Z Ran 1 test in 5.680s
2022-09-27T15:52:24.4933130Z 
2022-09-27T15:52:24.4933223Z OK
2022-09-27T15:52:24.4933362Z 
2022-09-27T15:52:24.4933503Z Generating XML reports...
2022-09-27T15:52:24.4971140Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155218.xml
2022-09-27T15:52:26.5587944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:26.5588451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:26.5589366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:26.5589819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:26.8156657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:26.8170929Z 
2022-09-27T15:52:26.8171119Z Running tests...
2022-09-27T15:52:26.8171563Z ----------------------------------------------------------------------
2022-09-27T15:52:28.4389196Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:28.4580230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5768
2022-09-27T15:52:28.4587247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5769
2022-09-27T15:52:30.1124431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:30.1124932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:30.1125529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:30.1125988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:30.1133610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:30.1134100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:30.1135518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:30.1136006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:30.3895786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:30.3916113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:31.8468993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctielmuj
2022-09-27T15:52:31.8469621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4t1irza
2022-09-27T15:52:31.8470153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctielmuj/_remote_module_non_scriptable.py
2022-09-27T15:52:31.8470714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4t1irza/_remote_module_non_scriptable.py
2022-09-27T15:52:32.3313737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:52:32.3314296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:52:32.8687022Z ok (6.051s)
2022-09-27T15:52:32.8687635Z 
2022-09-27T15:52:32.8688059Z ----------------------------------------------------------------------
2022-09-27T15:52:32.8688352Z Ran 1 test in 6.051s
2022-09-27T15:52:32.8688521Z 
2022-09-27T15:52:32.8688597Z OK
2022-09-27T15:52:32.8688744Z 
2022-09-27T15:52:32.8688880Z Generating XML reports...
2022-09-27T15:52:32.8725869Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155226.xml
2022-09-27T15:52:34.9659556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:34.9660136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:34.9660985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:34.9661480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:35.2189810Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:35.2204654Z 
2022-09-27T15:52:35.2221248Z Running tests...
2022-09-27T15:52:35.2221746Z ----------------------------------------------------------------------
2022-09-27T15:52:36.8420003Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:36.8610818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5883
2022-09-27T15:52:36.8618741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5884
2022-09-27T15:52:38.5159314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.5159837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.5167983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.5168524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.5343931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.5344386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.5355466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.5355960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.7946465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:38.8042828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:40.2663686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh9lrr11
2022-09-27T15:52:40.2664353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh9lrr11/_remote_module_non_scriptable.py
2022-09-27T15:52:40.2856290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkqdxghs
2022-09-27T15:52:40.2859205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkqdxghs/_remote_module_non_scriptable.py
2022-09-27T15:52:40.7504778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:52:40.7505352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:52:41.3721020Z ok (6.151s)
2022-09-27T15:52:41.3721284Z 
2022-09-27T15:52:41.3721714Z ----------------------------------------------------------------------
2022-09-27T15:52:41.3722065Z Ran 1 test in 6.151s
2022-09-27T15:52:41.3722215Z 
2022-09-27T15:52:41.3722315Z OK
2022-09-27T15:52:41.3722455Z 
2022-09-27T15:52:41.3722620Z Generating XML reports...
2022-09-27T15:52:41.3760751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155235.xml
2022-09-27T15:52:43.4516031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:43.4516982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:43.4517875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:43.4518342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:43.7107129Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:43.7121829Z 
2022-09-27T15:52:43.7122334Z Running tests...
2022-09-27T15:52:43.7122814Z ----------------------------------------------------------------------
2022-09-27T15:52:45.3389667Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:45.3581403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5998
2022-09-27T15:52:45.3589423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5999
2022-09-27T15:52:47.0316404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:47.0317002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:47.0324503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:47.0325001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:47.0522233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:47.0522688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:47.0533825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:47.0534365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:47.3097631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:47.3117133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:47.7649098Z skip: Need at least 4 CUDA devices (4.052s)
2022-09-27T15:52:47.7649479Z 
2022-09-27T15:52:47.7650219Z ----------------------------------------------------------------------
2022-09-27T15:52:47.7650580Z Ran 1 test in 4.053s
2022-09-27T15:52:47.7650760Z 
2022-09-27T15:52:47.7650878Z OK (skipped=1)
2022-09-27T15:52:47.7651044Z 
2022-09-27T15:52:47.7651153Z Generating XML reports...
2022-09-27T15:52:47.7690430Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155243.xml
2022-09-27T15:52:49.8343716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:49.8344252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:49.8345791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:49.8346273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:50.0893574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:50.0909630Z 
2022-09-27T15:52:50.0909815Z Running tests...
2022-09-27T15:52:50.0910454Z ----------------------------------------------------------------------
2022-09-27T15:52:51.7174770Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:51.7365083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6101
2022-09-27T15:52:51.7373123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6102
2022-09-27T15:52:53.4179513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.4180504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.4189484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.4190433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.4382448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.4382923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.4393497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.4393983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.6976156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:53.7094845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:54.1434575Z skip: Need at least 8 CUDA devices (4.052s)
2022-09-27T15:52:54.1434820Z 
2022-09-27T15:52:54.1435253Z ----------------------------------------------------------------------
2022-09-27T15:52:54.1435595Z Ran 1 test in 4.052s
2022-09-27T15:52:54.1437119Z 
2022-09-27T15:52:54.1437577Z OK (skipped=1)
2022-09-27T15:52:54.1437858Z 
2022-09-27T15:52:54.1438081Z Generating XML reports...
2022-09-27T15:52:54.1474833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155250.xml
2022-09-27T15:52:56.2334684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:56.2335257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:56.2337452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:56.2337956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:56.4896046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:52:56.4910257Z 
2022-09-27T15:52:56.4910441Z Running tests...
2022-09-27T15:52:56.4911438Z ----------------------------------------------------------------------
2022-09-27T15:52:58.1050099Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:58.1239532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6204
2022-09-27T15:52:58.1246870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6205
2022-09-27T15:52:59.7521427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.7521974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.7530080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.7530600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.7877260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.7877698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.7888687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.7889164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:00.0218769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:00.0452468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:00.0743089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfm2vsscs
2022-09-27T15:53:00.0745952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v0430qo
2022-09-27T15:53:00.0746539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfm2vsscs/_remote_module_non_scriptable.py
2022-09-27T15:53:00.0748630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v0430qo/_remote_module_non_scriptable.py
2022-09-27T15:53:00.0967625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:00.5306946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:00.5307408Z ok (4.039s)
2022-09-27T15:53:00.5307586Z 
2022-09-27T15:53:00.5308001Z ----------------------------------------------------------------------
2022-09-27T15:53:00.5308357Z Ran 1 test in 4.039s
2022-09-27T15:53:00.5308519Z 
2022-09-27T15:53:00.5308662Z OK
2022-09-27T15:53:00.5308807Z 
2022-09-27T15:53:00.5308913Z Generating XML reports...
2022-09-27T15:53:00.5346159Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155256.xml
2022-09-27T15:53:02.6518341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:02.6519197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:02.6520458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:02.6520950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:02.9219611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:02.9234619Z 
2022-09-27T15:53:02.9235041Z Running tests...
2022-09-27T15:53:02.9235549Z ----------------------------------------------------------------------
2022-09-27T15:53:04.5703162Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:04.5894604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6317
2022-09-27T15:53:04.5901895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6318
2022-09-27T15:53:06.2438842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.2439289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.2439900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.2440363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.2448188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.2448793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.2449390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.2450282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.5219081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:06.5219575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:06.5541701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejo2zbqu
2022-09-27T15:53:06.5542245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ankkcbl
2022-09-27T15:53:06.5544930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejo2zbqu/_remote_module_non_scriptable.py
2022-09-27T15:53:06.5545514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ankkcbl/_remote_module_non_scriptable.py
2022-09-27T15:53:06.5757245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:06.5757754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:06.9962066Z ok (4.072s)
2022-09-27T15:53:06.9962350Z 
2022-09-27T15:53:06.9962761Z ----------------------------------------------------------------------
2022-09-27T15:53:06.9963103Z Ran 1 test in 4.073s
2022-09-27T15:53:06.9963279Z 
2022-09-27T15:53:06.9963336Z OK
2022-09-27T15:53:06.9963440Z 
2022-09-27T15:53:06.9963585Z Generating XML reports...
2022-09-27T15:53:07.0001696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155302.xml
2022-09-27T15:53:09.0728045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:09.0728594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:09.0730517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:09.0731256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:09.3289095Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:09.3303268Z 
2022-09-27T15:53:09.3303514Z Running tests...
2022-09-27T15:53:09.3304476Z ----------------------------------------------------------------------
2022-09-27T15:53:09.3317905Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-09-27T15:53:10.9792145Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:10.9982474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6430
2022-09-27T15:53:10.9989523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6431
2022-09-27T15:53:12.6890893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:12.6891442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:12.6898903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:12.6899368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:12.7158373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:12.7158813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:12.7169877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:12.7170578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:12.9560552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:12.9744240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:13.0065755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzg1bsw7
2022-09-27T15:53:13.0068386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzg1bsw7/_remote_module_non_scriptable.py
2022-09-27T15:53:13.0069003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpira83yby
2022-09-27T15:53:13.0071365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpira83yby/_remote_module_non_scriptable.py
2022-09-27T15:53:13.0316112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:13.0316634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:13.5050921Z ok (4.174s)
2022-09-27T15:53:13.5051179Z 
2022-09-27T15:53:13.5051933Z ----------------------------------------------------------------------
2022-09-27T15:53:13.5052304Z Ran 1 test in 4.175s
2022-09-27T15:53:13.5052453Z 
2022-09-27T15:53:13.5052556Z OK
2022-09-27T15:53:13.5052695Z 
2022-09-27T15:53:13.5052837Z Generating XML reports...
2022-09-27T15:53:13.5089865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155309.xml
2022-09-27T15:53:15.6051201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:15.6051760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:15.6054334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:15.6054797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:15.8617840Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:15.8633139Z 
2022-09-27T15:53:15.8633669Z Running tests...
2022-09-27T15:53:15.8634197Z ----------------------------------------------------------------------
2022-09-27T15:53:15.8649367Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-09-27T15:53:17.4931599Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:17.5127051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6573
2022-09-27T15:53:17.5134696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6574
2022-09-27T15:53:19.1698382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:19.1699248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:19.1707691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:19.1708161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:19.1899527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:19.1900003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:19.1910086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:19.1910579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:19.4482944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:19.4553240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:19.4802313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyccqwql4
2022-09-27T15:53:19.4804733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y8z2pcv
2022-09-27T15:53:19.4805263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyccqwql4/_remote_module_non_scriptable.py
2022-09-27T15:53:19.4807673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y8z2pcv/_remote_module_non_scriptable.py
2022-09-27T15:53:19.9196008Z ok (4.056s)
2022-09-27T15:53:19.9196431Z 
2022-09-27T15:53:19.9196891Z ----------------------------------------------------------------------
2022-09-27T15:53:19.9197223Z Ran 1 test in 4.056s
2022-09-27T15:53:19.9197372Z 
2022-09-27T15:53:19.9197452Z OK
2022-09-27T15:53:19.9197596Z 
2022-09-27T15:53:19.9197737Z Generating XML reports...
2022-09-27T15:53:19.9235656Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155315.xml
2022-09-27T15:53:22.0145161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:22.0146110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:22.0146978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:22.0147483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:22.3058556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:22.3073200Z 
2022-09-27T15:53:22.3073952Z Running tests...
2022-09-27T15:53:22.3074408Z ----------------------------------------------------------------------
2022-09-27T15:53:23.9277063Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:23.9471959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6716
2022-09-27T15:53:23.9480481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6717
2022-09-27T15:53:25.5768237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:25.5768777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:25.5777155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:25.5777661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:25.6061336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:25.6061828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:25.6072922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:25.6073717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:25.8435199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:25.8440080Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8441524Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8442827Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8444164Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8445243Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8446326Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8630600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:25.8636465Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8637987Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8639314Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8640394Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8641644Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:25.8642717Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T15:53:26.2537729Z ok (3.946s)
2022-09-27T15:53:26.2537933Z 
2022-09-27T15:53:26.2538485Z ----------------------------------------------------------------------
2022-09-27T15:53:26.2538871Z Ran 1 test in 3.946s
2022-09-27T15:53:26.2539051Z 
2022-09-27T15:53:26.2539152Z OK
2022-09-27T15:53:26.2539297Z 
2022-09-27T15:53:26.2539440Z Generating XML reports...
2022-09-27T15:53:26.2578197Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155322.xml
2022-09-27T15:53:28.3674148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:28.3674689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:28.3675735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:28.3676210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:28.6359535Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:28.6375919Z 
2022-09-27T15:53:28.6376187Z Running tests...
2022-09-27T15:53:28.6376625Z ----------------------------------------------------------------------
2022-09-27T15:53:30.2865615Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:30.3056497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6819
2022-09-27T15:53:30.3063937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6820
2022-09-27T15:53:31.9371086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:31.9371617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:31.9379421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:31.9379918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:31.9762230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:31.9762688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:31.9774026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:31.9774520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:32.2010189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:32.2261073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:32.2679908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:32.2680429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:32.2681277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:32.2681967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:33.6928108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszmo0lfi
2022-09-27T15:53:33.6928786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszmo0lfi/_remote_module_non_scriptable.py
2022-09-27T15:53:33.7028025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfce0pw3o
2022-09-27T15:53:33.7031293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfce0pw3o/_remote_module_non_scriptable.py
2022-09-27T15:53:34.1755942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:34.1756513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:34.1896321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:34.1896797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:34.7183039Z ok (6.080s)
2022-09-27T15:53:34.7183314Z 
2022-09-27T15:53:34.7183669Z ----------------------------------------------------------------------
2022-09-27T15:53:34.7184031Z Ran 1 test in 6.081s
2022-09-27T15:53:34.7184215Z 
2022-09-27T15:53:34.7184316Z OK
2022-09-27T15:53:34.7184453Z 
2022-09-27T15:53:34.7184588Z Generating XML reports...
2022-09-27T15:53:34.7222522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155328.xml
2022-09-27T15:53:36.8422100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:36.8422600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:36.8423434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:36.8423924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:37.1001411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:37.1016947Z 
2022-09-27T15:53:37.1017215Z Running tests...
2022-09-27T15:53:37.1017619Z ----------------------------------------------------------------------
2022-09-27T15:53:38.7109020Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:38.7300939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6934
2022-09-27T15:53:38.7308314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6935
2022-09-27T15:53:40.3819534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:40.3828134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:40.3828743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:40.3829264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:40.4039867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:40.4040367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:40.4051193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:40.4051687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:40.6598817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:40.6689357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:40.6925215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yp0sj6y
2022-09-27T15:53:40.6926195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo_w8b3h
2022-09-27T15:53:40.6927745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yp0sj6y/_remote_module_non_scriptable.py
2022-09-27T15:53:40.6929140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo_w8b3h/_remote_module_non_scriptable.py
2022-09-27T15:53:41.1367279Z ok (4.035s)
2022-09-27T15:53:41.1367426Z 
2022-09-27T15:53:41.1368099Z ----------------------------------------------------------------------
2022-09-27T15:53:41.1368817Z Ran 1 test in 4.035s
2022-09-27T15:53:41.1369072Z 
2022-09-27T15:53:41.1369175Z OK
2022-09-27T15:53:41.1369321Z 
2022-09-27T15:53:41.1369458Z Generating XML reports...
2022-09-27T15:53:41.1407675Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155337.xml
2022-09-27T15:53:43.2794745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:43.2795339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:43.2796662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:43.2797428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:43.5479958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:43.5494451Z 
2022-09-27T15:53:43.5494767Z Running tests...
2022-09-27T15:53:43.5495715Z ----------------------------------------------------------------------
2022-09-27T15:53:45.1648873Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:45.1841005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7077
2022-09-27T15:53:45.1847942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7078
2022-09-27T15:53:46.8213687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:46.8214289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:46.8222344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:46.8223154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:46.8591211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:46.8591659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:46.8602912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:46.8603341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:47.0923046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:47.1182799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:47.1410287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_94jg9x7
2022-09-27T15:53:47.1411322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp05ov6t2
2022-09-27T15:53:47.1412324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_94jg9x7/_remote_module_non_scriptable.py
2022-09-27T15:53:47.1415189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp05ov6t2/_remote_module_non_scriptable.py
2022-09-27T15:53:47.5907225Z ok (4.041s)
2022-09-27T15:53:47.5907407Z 
2022-09-27T15:53:47.5907858Z ----------------------------------------------------------------------
2022-09-27T15:53:47.5908171Z Ran 1 test in 4.041s
2022-09-27T15:53:47.5908351Z 
2022-09-27T15:53:47.5908454Z OK
2022-09-27T15:53:47.5908599Z 
2022-09-27T15:53:47.5908744Z Generating XML reports...
2022-09-27T15:53:47.5947118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155343.xml
2022-09-27T15:53:49.6787361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.6787913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.6789364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.6789882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.9473620Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:49.9489240Z 
2022-09-27T15:53:49.9489728Z Running tests...
2022-09-27T15:53:49.9490447Z ----------------------------------------------------------------------
2022-09-27T15:53:51.5817847Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:51.6010716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7220
2022-09-27T15:53:51.6018650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7221
2022-09-27T15:53:53.2576756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:53.2577718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:53.2588225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:53.2589052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:53.2893858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:53.2894292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:53.2905278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:53.2905730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:53.5241885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:53.5468548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:55.0068759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhk0bui7
2022-09-27T15:53:55.0069436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhk0bui7/_remote_module_non_scriptable.py
2022-09-27T15:53:55.0395672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w92v019
2022-09-27T15:53:55.0396783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w92v019/_remote_module_non_scriptable.py
2022-09-27T15:53:56.4023636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:56.4024267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:53:57.2144729Z ok (7.265s)
2022-09-27T15:53:57.2145207Z 
2022-09-27T15:53:57.2146026Z ----------------------------------------------------------------------
2022-09-27T15:53:57.2146486Z Ran 1 test in 7.265s
2022-09-27T15:53:57.2146616Z 
2022-09-27T15:53:57.2146695Z OK
2022-09-27T15:53:57.2146842Z 
2022-09-27T15:53:57.2151001Z Generating XML reports...
2022-09-27T15:53:57.2184814Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155349.xml
2022-09-27T15:53:59.3589665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:59.3590228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:59.3591358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:59.3592123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:59.6295352Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:53:59.6311075Z 
2022-09-27T15:53:59.6311399Z Running tests...
2022-09-27T15:53:59.6311867Z ----------------------------------------------------------------------
2022-09-27T15:54:01.2746363Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:01.2944760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7335
2022-09-27T15:54:01.2952123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7336
2022-09-27T15:54:02.9382377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:02.9382897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:02.9390459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:02.9390959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:02.9752193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:02.9752667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:02.9763711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:02.9764206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:03.2095285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:03.2333764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:04.7045302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkht19d2
2022-09-27T15:54:04.7045965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkht19d2/_remote_module_non_scriptable.py
2022-09-27T15:54:04.7261021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmltxtnfs
2022-09-27T15:54:04.7263823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmltxtnfs/_remote_module_non_scriptable.py
2022-09-27T15:54:05.4132808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:54:05.4133376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:54:06.0058507Z ok (6.374s)
2022-09-27T15:54:06.0058825Z 
2022-09-27T15:54:06.0059321Z ----------------------------------------------------------------------
2022-09-27T15:54:06.0059693Z Ran 1 test in 6.375s
2022-09-27T15:54:06.0059806Z 
2022-09-27T15:54:06.0059884Z OK
2022-09-27T15:54:06.0059998Z 
2022-09-27T15:54:06.0060144Z Generating XML reports...
2022-09-27T15:54:06.0098913Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155359.xml
2022-09-27T15:54:08.1115874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:08.1116812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:08.1118552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:08.1119321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:08.3796165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:08.3811938Z 
2022-09-27T15:54:08.3812554Z Running tests...
2022-09-27T15:54:08.3813210Z ----------------------------------------------------------------------
2022-09-27T15:54:10.0357832Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:10.0549110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7450
2022-09-27T15:54:10.0557223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7451
2022-09-27T15:54:10.0563782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7452
2022-09-27T15:54:10.0570549Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7453
2022-09-27T15:54:11.7215227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.7215738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.7223992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.7224496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.7341243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.7341693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.7352933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.7353474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.7589776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.7590262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.7600797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.7601373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.7678453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.7678924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.7690620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.7691418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:12.0140712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:12.0142892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:12.0201513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:12.0360149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:12.6642180Z ok (4.283s)
2022-09-27T15:54:12.6642430Z 
2022-09-27T15:54:12.6642877Z ----------------------------------------------------------------------
2022-09-27T15:54:12.6643172Z Ran 1 test in 4.283s
2022-09-27T15:54:12.6643345Z 
2022-09-27T15:54:12.6643446Z OK
2022-09-27T15:54:12.6643566Z 
2022-09-27T15:54:12.6643710Z Generating XML reports...
2022-09-27T15:54:12.6682730Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155408.xml
2022-09-27T15:54:14.7912067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:14.7913112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:14.7914280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:14.7915213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:15.0631403Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:15.0648122Z 
2022-09-27T15:54:15.0648730Z Running tests...
2022-09-27T15:54:15.0649612Z ----------------------------------------------------------------------
2022-09-27T15:54:16.7129524Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:16.7329038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7633
2022-09-27T15:54:16.7338170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7634
2022-09-27T15:54:16.7345224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7635
2022-09-27T15:54:16.7352120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7636
2022-09-27T15:54:18.4005386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4005936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4014287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4014797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.4257182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4257695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4268896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4269403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.4646445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4646991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4647553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4648049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4657953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4658788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.4659382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4659866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.6781265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:18.6911715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:18.7343639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:18.7398004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:21.2466904Z ok (6.181s)
2022-09-27T15:54:21.2467297Z 
2022-09-27T15:54:21.2467964Z ----------------------------------------------------------------------
2022-09-27T15:54:21.2468552Z Ran 1 test in 6.182s
2022-09-27T15:54:21.2468847Z 
2022-09-27T15:54:21.2468998Z OK
2022-09-27T15:54:21.2469232Z 
2022-09-27T15:54:21.2469456Z Generating XML reports...
2022-09-27T15:54:21.2508932Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155415.xml
2022-09-27T15:54:23.3513931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:23.3514464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:23.3515744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:23.3516235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:23.6177090Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:23.6192015Z 
2022-09-27T15:54:23.6192432Z Running tests...
2022-09-27T15:54:23.6192893Z ----------------------------------------------------------------------
2022-09-27T15:54:25.2826126Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:25.3024005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7820
2022-09-27T15:54:25.3031556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7821
2022-09-27T15:54:25.3038298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7822
2022-09-27T15:54:25.3045288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7823
2022-09-27T15:54:27.0070400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:27.0070958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:27.0081659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:27.0082178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:27.0395655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:27.0406168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:27.0406800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:27.0407285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:27.0410685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:27.0411156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:27.0423802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:27.0424642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:27.0515522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:27.0515963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:27.0527036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:27.0527527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:27.3029033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:27.3174825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:27.3218388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:27.3218904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:28.0122034Z ok (4.393s)
2022-09-27T15:54:28.0122270Z 
2022-09-27T15:54:28.0122695Z ----------------------------------------------------------------------
2022-09-27T15:54:28.0123046Z Ran 1 test in 4.393s
2022-09-27T15:54:28.0123222Z 
2022-09-27T15:54:28.0123320Z OK
2022-09-27T15:54:28.0123466Z 
2022-09-27T15:54:28.0123591Z Generating XML reports...
2022-09-27T15:54:28.0161418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155423.xml
2022-09-27T15:54:30.1054695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:30.1055221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:30.1057683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:30.1058200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:30.3692132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:30.3706677Z 
2022-09-27T15:54:30.3706832Z Running tests...
2022-09-27T15:54:30.3707524Z ----------------------------------------------------------------------
2022-09-27T15:54:31.9916534Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:32.0107783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8003
2022-09-27T15:54:32.0114672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8004
2022-09-27T15:54:32.0121024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8005
2022-09-27T15:54:32.0127502Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8006
2022-09-27T15:54:33.6897537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:33.6898097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:33.6902701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:33.6903139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:33.6906250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:33.6906706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:33.6913837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:33.6914321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:33.7213561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:33.7214303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:33.7225421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:33.7225921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:33.7299504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:33.7299966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:33.7310996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:33.7311489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:33.9688059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:33.9810058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:33.9888652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:33.9998331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:34.0313847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:34.0314367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:54:34.0417967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:34.0418478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T15:54:34.0419492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:54:34.0420295Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:54:34.0519018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:54:34.0519698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:54:34.6199989Z ok (4.249s)
2022-09-27T15:54:34.6200221Z 
2022-09-27T15:54:34.6200617Z ----------------------------------------------------------------------
2022-09-27T15:54:34.6200962Z Ran 1 test in 4.249s
2022-09-27T15:54:34.6201144Z 
2022-09-27T15:54:34.6201238Z OK
2022-09-27T15:54:34.6201364Z 
2022-09-27T15:54:34.6201502Z Generating XML reports...
2022-09-27T15:54:34.6240651Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155430.xml
2022-09-27T15:54:36.7342147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:36.7342762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:36.7344489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:36.7344965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:36.9950563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:36.9965751Z 
2022-09-27T15:54:36.9965989Z Running tests...
2022-09-27T15:54:36.9966425Z ----------------------------------------------------------------------
2022-09-27T15:54:38.6210310Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:38.6403875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8186
2022-09-27T15:54:38.6412805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8187
2022-09-27T15:54:38.6419791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8188
2022-09-27T15:54:38.6427086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8189
2022-09-27T15:54:40.3142570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.3143046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.3151187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.3151680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.3218788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.3219278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.3229701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.3230203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.3455962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.3456423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.3468064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.3468547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.3534042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.3534744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.3545478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.3545970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.5987764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:40.6058202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:40.6115383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:40.6241291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:41.2499832Z ok (4.253s)
2022-09-27T15:54:41.2500074Z 
2022-09-27T15:54:41.2500502Z ----------------------------------------------------------------------
2022-09-27T15:54:41.2500861Z Ran 1 test in 4.253s
2022-09-27T15:54:41.2501033Z 
2022-09-27T15:54:41.2501135Z OK
2022-09-27T15:54:41.2501299Z 
2022-09-27T15:54:41.2501440Z Generating XML reports...
2022-09-27T15:54:41.2540299Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155436.xml
2022-09-27T15:54:43.3681238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:43.3681719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:43.3683040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:43.3683720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:43.6267359Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:43.6282068Z 
2022-09-27T15:54:43.6282327Z Running tests...
2022-09-27T15:54:43.6282992Z ----------------------------------------------------------------------
2022-09-27T15:54:45.2487673Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:45.2679468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8369
2022-09-27T15:54:45.2686683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8370
2022-09-27T15:54:45.2692951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8371
2022-09-27T15:54:45.2699750Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8372
2022-09-27T15:54:46.9643859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:46.9644374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:46.9653047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:46.9653825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:46.9705898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:46.9706633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:46.9716949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:46.9717702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.0170892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.0171396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.0181136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.0181881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.0391037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.0391556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.0403779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.0404542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.2312402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:47.2547064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:47.2791471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:47.3045816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:47.8773104Z ok (4.249s)
2022-09-27T15:54:47.8773373Z 
2022-09-27T15:54:47.8773819Z ----------------------------------------------------------------------
2022-09-27T15:54:47.8774174Z Ran 1 test in 4.249s
2022-09-27T15:54:47.8774342Z 
2022-09-27T15:54:47.8774440Z OK
2022-09-27T15:54:47.8774580Z 
2022-09-27T15:54:47.8774718Z Generating XML reports...
2022-09-27T15:54:47.8812984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155443.xml
2022-09-27T15:54:49.9905989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:49.9906848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:49.9908159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:49.9908619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:50.2609476Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:50.2625539Z 
2022-09-27T15:54:50.2626103Z Running tests...
2022-09-27T15:54:50.2626743Z ----------------------------------------------------------------------
2022-09-27T15:54:51.9320131Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:51.9520011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8552
2022-09-27T15:54:51.9527682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8553
2022-09-27T15:54:51.9534779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8554
2022-09-27T15:54:51.9542123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8555
2022-09-27T15:54:53.6915985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.6916582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.6924505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.6924969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.7349899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.7350387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.7357029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.7357463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.7360857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.7361363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.7368574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.7369098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.7623711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.7624138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.7635375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.7635807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.9847028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:54.0054211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:54:54.0128562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:54.0279887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:54:55.1629309Z ok (4.900s)
2022-09-27T15:54:55.1629546Z 
2022-09-27T15:54:55.1629972Z ----------------------------------------------------------------------
2022-09-27T15:54:55.1630343Z Ran 1 test in 4.900s
2022-09-27T15:54:55.1630497Z 
2022-09-27T15:54:55.1630587Z OK
2022-09-27T15:54:55.1630728Z 
2022-09-27T15:54:55.1630864Z Generating XML reports...
2022-09-27T15:54:55.1671732Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155450.xml
2022-09-27T15:54:57.3230259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:57.3230755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:57.3233108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:57.3233600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:57.5901406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:54:57.5917007Z 
2022-09-27T15:54:57.5917412Z Running tests...
2022-09-27T15:54:57.5917864Z ----------------------------------------------------------------------
2022-09-27T15:54:59.2505865Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:59.2704571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8759
2022-09-27T15:54:59.2712125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8760
2022-09-27T15:54:59.2719018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8761
2022-09-27T15:54:59.2725878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8762
2022-09-27T15:55:00.9566693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:00.9567292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:00.9576399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:00.9576905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:00.9586397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:00.9586879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:00.9599216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:00.9599711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:00.9632862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:00.9633381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:00.9646072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:00.9646567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:00.9730336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:00.9730824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:00.9744148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:00.9744638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:01.2438393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:01.2439569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:01.2617308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:01.2660644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:05.4873183Z ok (7.895s)
2022-09-27T15:55:05.4873432Z 
2022-09-27T15:55:05.4873872Z ----------------------------------------------------------------------
2022-09-27T15:55:05.4874261Z Ran 1 test in 7.895s
2022-09-27T15:55:05.4874412Z 
2022-09-27T15:55:05.4874514Z OK
2022-09-27T15:55:05.4874661Z 
2022-09-27T15:55:05.4874805Z Generating XML reports...
2022-09-27T15:55:05.4913219Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155457.xml
2022-09-27T15:55:07.5245569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:07.5246624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:07.5247809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:07.5249224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:07.7862372Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:07.7876291Z 
2022-09-27T15:55:07.7876546Z Running tests...
2022-09-27T15:55:07.7877004Z ----------------------------------------------------------------------
2022-09-27T15:55:09.3955436Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:09.4144732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8970
2022-09-27T15:55:09.4153227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8971
2022-09-27T15:55:09.4160478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8972
2022-09-27T15:55:09.4166851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8973
2022-09-27T15:55:11.0485670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.0486169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.0494700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.0495217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.0823466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.0824438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.0835038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.0836411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.1061152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.1062123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.1072690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.1073650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.1214645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.1215634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.1226339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.1227337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.3396332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:11.3544896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:11.3688437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:11.3858222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:12.0240040Z ok (4.236s)
2022-09-27T15:55:12.0240269Z 
2022-09-27T15:55:12.0240725Z ----------------------------------------------------------------------
2022-09-27T15:55:12.0241004Z Ran 1 test in 4.236s
2022-09-27T15:55:12.0241178Z 
2022-09-27T15:55:12.0241313Z OK
2022-09-27T15:55:12.0241463Z 
2022-09-27T15:55:12.0241622Z Generating XML reports...
2022-09-27T15:55:12.0282850Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155507.xml
2022-09-27T15:55:14.1728914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:14.1730129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:14.1731190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:14.1731682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:14.4363578Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:14.4379076Z 
2022-09-27T15:55:14.4379223Z Running tests...
2022-09-27T15:55:14.4379954Z ----------------------------------------------------------------------
2022-09-27T15:55:16.0585556Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:16.0778660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9153
2022-09-27T15:55:16.0785802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9154
2022-09-27T15:55:16.0792612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9155
2022-09-27T15:55:16.0799903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9156
2022-09-27T15:55:17.7587419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:17.7587942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:17.7596476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:17.7596978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:17.7951517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:17.7952307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:17.7962663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:17.7963174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:17.7977400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:17.7977848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:17.7988723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:17.7989218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:17.8260933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:17.8261387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:17.8271619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:17.8272117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:18.0538013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:18.0628817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:18.0808885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:18.0826804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:20.5915693Z ok (6.153s)
2022-09-27T15:55:20.5916156Z 
2022-09-27T15:55:20.5916813Z ----------------------------------------------------------------------
2022-09-27T15:55:20.5917416Z Ran 1 test in 6.154s
2022-09-27T15:55:20.5917698Z 
2022-09-27T15:55:20.5917818Z OK
2022-09-27T15:55:20.5918082Z 
2022-09-27T15:55:20.5918306Z Generating XML reports...
2022-09-27T15:55:20.5957386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155514.xml
2022-09-27T15:55:22.7046348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:22.7046930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:22.7047835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:22.7048322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:22.9635086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:22.9650478Z 
2022-09-27T15:55:22.9651124Z Running tests...
2022-09-27T15:55:22.9651632Z ----------------------------------------------------------------------
2022-09-27T15:55:24.5705258Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:24.5897239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9340
2022-09-27T15:55:24.5905048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9341
2022-09-27T15:55:24.5912120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9342
2022-09-27T15:55:24.5919000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9343
2022-09-27T15:55:26.2777875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:26.2778428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:26.2787638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:26.2788445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:26.3136108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:26.3136613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:26.3147222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:26.3147691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:26.3330730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:26.3331197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:26.3342607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:26.3343076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:26.3834251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:26.3834767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:26.3847395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:26.3847869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:26.5708696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:26.5806332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:26.5957808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:26.6447643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:29.1032023Z ok (6.138s)
2022-09-27T15:55:29.1032319Z 
2022-09-27T15:55:29.1032786Z ----------------------------------------------------------------------
2022-09-27T15:55:29.1033162Z Ran 1 test in 6.138s
2022-09-27T15:55:29.1033705Z 
2022-09-27T15:55:29.1033804Z OK
2022-09-27T15:55:29.1033920Z 
2022-09-27T15:55:29.1034063Z Generating XML reports...
2022-09-27T15:55:29.1072964Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155522.xml
2022-09-27T15:55:31.2146766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:31.2147266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:31.2148570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:31.2149013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:31.4753325Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:31.4768024Z 
2022-09-27T15:55:31.4768587Z Running tests...
2022-09-27T15:55:31.4769117Z ----------------------------------------------------------------------
2022-09-27T15:55:33.0687796Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:33.0879213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9527
2022-09-27T15:55:33.0887090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9528
2022-09-27T15:55:33.0894225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9529
2022-09-27T15:55:33.0901175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9530
2022-09-27T15:55:34.7749179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:34.7749715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:34.7758355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:34.7758892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.7759503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:34.7759965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:34.7771823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:34.7772314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.7779599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:34.7780069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:34.7791838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:34.7792333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.8042207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:34.8042668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:34.8053074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:34.8053555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:35.0598873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:35.0603958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:35.0651341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:35.0660255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:35.6973867Z ok (4.220s)
2022-09-27T15:55:35.6974167Z 
2022-09-27T15:55:35.6974569Z ----------------------------------------------------------------------
2022-09-27T15:55:35.6974909Z Ran 1 test in 4.221s
2022-09-27T15:55:35.6975084Z 
2022-09-27T15:55:35.6975183Z OK
2022-09-27T15:55:35.6976136Z 
2022-09-27T15:55:35.6976300Z Generating XML reports...
2022-09-27T15:55:35.7014985Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155531.xml
2022-09-27T15:55:37.7968884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:37.7969416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:37.7970666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:37.7971134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:38.0532643Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:38.0548023Z 
2022-09-27T15:55:38.0548169Z Running tests...
2022-09-27T15:55:38.0548936Z ----------------------------------------------------------------------
2022-09-27T15:55:39.6884166Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:39.7076648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9710
2022-09-27T15:55:39.7084282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9711
2022-09-27T15:55:39.7090572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9712
2022-09-27T15:55:39.7097332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9713
2022-09-27T15:55:41.4235158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:41.4235752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:41.4243607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:41.4244095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.4592725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:41.4593207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:41.4603292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:41.4603784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.4751493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:41.4751981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:41.4763426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:41.4763916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.4768607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:41.4769080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:41.4780628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:41.4781092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.7219039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:41.7337846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:41.7450760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:41.7555464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:42.4173046Z ok (4.362s)
2022-09-27T15:55:42.4173356Z 
2022-09-27T15:55:42.4173802Z ----------------------------------------------------------------------
2022-09-27T15:55:42.4176095Z Ran 1 test in 4.362s
2022-09-27T15:55:42.4176294Z 
2022-09-27T15:55:42.4176402Z OK
2022-09-27T15:55:42.4176549Z 
2022-09-27T15:55:42.4176665Z Generating XML reports...
2022-09-27T15:55:42.4214508Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155538.xml
2022-09-27T15:55:44.5468808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:44.5469396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:44.5470411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:44.5470924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:44.8153332Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:44.8168618Z 
2022-09-27T15:55:44.8168866Z Running tests...
2022-09-27T15:55:44.8169340Z ----------------------------------------------------------------------
2022-09-27T15:55:46.4675358Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:46.4872966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9893
2022-09-27T15:55:46.4880617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9894
2022-09-27T15:55:46.4888399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9895
2022-09-27T15:55:46.4895118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9896
2022-09-27T15:55:48.1471072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.1471593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.1480026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.1480521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.1849044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.1849487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.1850675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.1851170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.1860600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.1861090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.1861993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.1862475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.1913304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.1913773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.1926658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.1927153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.4468535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:48.4658576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:48.4665740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:48.4669347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:48.4872491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:48.4974464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:48.4982990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T15:55:48.4983572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T15:55:48.4984377Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:55:48.4985106Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:55:48.5076961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:55:48.5077634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T15:55:49.0965050Z ok (4.279s)
2022-09-27T15:55:49.0965332Z 
2022-09-27T15:55:49.0965744Z ----------------------------------------------------------------------
2022-09-27T15:55:49.0966114Z Ran 1 test in 4.279s
2022-09-27T15:55:49.0966276Z 
2022-09-27T15:55:49.0966376Z OK
2022-09-27T15:55:49.0966469Z 
2022-09-27T15:55:49.0966608Z Generating XML reports...
2022-09-27T15:55:49.1005230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155544.xml
2022-09-27T15:55:51.2027296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.2027832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.2029082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.2029577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.4742000Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:51.4756898Z 
2022-09-27T15:55:51.4757274Z Running tests...
2022-09-27T15:55:51.4757736Z ----------------------------------------------------------------------
2022-09-27T15:55:53.1364797Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:53.1563261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10076
2022-09-27T15:55:53.1570854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10077
2022-09-27T15:55:53.1578310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10078
2022-09-27T15:55:53.1585350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10079
2022-09-27T15:55:54.8499536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.8500034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.8515064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.8515562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:54.8630564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.8631032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.8642749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.8643239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:54.8701234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.8701684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.8713561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.8714059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:54.9023472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.9023958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.9035013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.9035517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.1513156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:55:55.1516582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:55:55.1592023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:55.1654872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:55.7659678Z ok (4.290s)
2022-09-27T15:55:55.7660130Z 
2022-09-27T15:55:55.7660867Z ----------------------------------------------------------------------
2022-09-27T15:55:55.7661503Z Ran 1 test in 4.290s
2022-09-27T15:55:55.7661700Z 
2022-09-27T15:55:55.7661841Z OK
2022-09-27T15:55:55.7661983Z 
2022-09-27T15:55:55.7662137Z Generating XML reports...
2022-09-27T15:55:55.7698938Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155551.xml
2022-09-27T15:55:57.8317253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:57.8318177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:57.8319543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:57.8320064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:58.0892885Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:55:58.0908042Z 
2022-09-27T15:55:58.0908237Z Running tests...
2022-09-27T15:55:58.0909199Z ----------------------------------------------------------------------
2022-09-27T15:55:59.7278301Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:59.7473531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10259
2022-09-27T15:55:59.7481876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10260
2022-09-27T15:55:59.7489212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10261
2022-09-27T15:55:59.7496117Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10262
2022-09-27T15:56:01.4522143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.4522690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.4532743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.4533211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:01.4923783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.4924320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.4935215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.4935686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:01.5083558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.5084035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.5095930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.5096606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:01.5168861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.5169351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.5182361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.5182831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:01.7640329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:01.7728603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:01.7746354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:01.7847328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:02.3576882Z ok (4.266s)
2022-09-27T15:56:02.3577131Z 
2022-09-27T15:56:02.3577557Z ----------------------------------------------------------------------
2022-09-27T15:56:02.3577938Z Ran 1 test in 4.267s
2022-09-27T15:56:02.3578107Z 
2022-09-27T15:56:02.3578211Z OK
2022-09-27T15:56:02.3578350Z 
2022-09-27T15:56:02.3578494Z Generating XML reports...
2022-09-27T15:56:02.3616130Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155558.xml
2022-09-27T15:56:04.4891883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.4892426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.4894164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:04.4894660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:04.7585586Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:04.7601934Z 
2022-09-27T15:56:04.7602057Z Running tests...
2022-09-27T15:56:04.7602785Z ----------------------------------------------------------------------
2022-09-27T15:56:06.4342875Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:06.4543216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10442
2022-09-27T15:56:06.4551243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10443
2022-09-27T15:56:06.4558251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10444
2022-09-27T15:56:06.4565662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10445
2022-09-27T15:56:08.1403900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.1404418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.1405017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.1405801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.1406364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.1406831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.1414400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.1414884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.1415459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.1415940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.1418231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.1418718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.1684023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:08.1684510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:08.1696557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:08.1697041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:08.4176151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:08.4347419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:08.4349520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:08.4461671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:10.8678688Z ok (6.107s)
2022-09-27T15:56:10.8678970Z 
2022-09-27T15:56:10.8679422Z ----------------------------------------------------------------------
2022-09-27T15:56:10.8679779Z Ran 1 test in 6.108s
2022-09-27T15:56:10.8679952Z 
2022-09-27T15:56:10.8680060Z OK
2022-09-27T15:56:10.8680179Z 
2022-09-27T15:56:10.8680319Z Generating XML reports...
2022-09-27T15:56:10.8719438Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155604.xml
2022-09-27T15:56:12.9689051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:12.9689840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:12.9691083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:12.9691580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.2350139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:13.2366661Z 
2022-09-27T15:56:13.2367057Z Running tests...
2022-09-27T15:56:13.2367669Z ----------------------------------------------------------------------
2022-09-27T15:56:14.8854268Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:14.9044314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10629
2022-09-27T15:56:14.9052890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10630
2022-09-27T15:56:14.9059965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10631
2022-09-27T15:56:14.9066683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10632
2022-09-27T15:56:16.6086614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:16.6087493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:16.6099490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:16.6099985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:16.6182457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:16.6182880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:16.6196890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:16.6197405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:16.6458034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:16.6458512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:16.6469518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:16.6470019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:16.6648880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:16.6649380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:16.6661395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:16.6661937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:16.9007869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:16.9143044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:16.9148983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:16.9241590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:17.9145211Z ok (4.677s)
2022-09-27T15:56:17.9145461Z 
2022-09-27T15:56:17.9145899Z ----------------------------------------------------------------------
2022-09-27T15:56:17.9146238Z Ran 1 test in 4.678s
2022-09-27T15:56:17.9146423Z 
2022-09-27T15:56:17.9146514Z OK
2022-09-27T15:56:17.9146663Z 
2022-09-27T15:56:17.9146813Z Generating XML reports...
2022-09-27T15:56:17.9187091Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155613.xml
2022-09-27T15:56:19.9792108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:19.9792961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:19.9794885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:19.9795424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:20.2487311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:20.2503799Z 
2022-09-27T15:56:20.2504243Z Running tests...
2022-09-27T15:56:20.2504801Z ----------------------------------------------------------------------
2022-09-27T15:56:21.8998394Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:21.9313005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10836
2022-09-27T15:56:21.9319973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10837
2022-09-27T15:56:21.9326981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10838
2022-09-27T15:56:21.9335220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10839
2022-09-27T15:56:23.6169102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:23.6169871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:23.6178304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:23.6178801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.6473791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:23.6474339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:23.6484446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:23.6484980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.6569437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:23.6570120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:23.6580776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:23.6581302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.6885640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:23.6886158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:23.6898393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:23.6898890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.9109766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:23.9225047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:23.9281896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:23.9511441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:24.7411901Z ok (4.490s)
2022-09-27T15:56:24.7412164Z 
2022-09-27T15:56:24.7412581Z ----------------------------------------------------------------------
2022-09-27T15:56:24.7412965Z Ran 1 test in 4.491s
2022-09-27T15:56:24.7413109Z 
2022-09-27T15:56:24.7413205Z OK
2022-09-27T15:56:24.7413346Z 
2022-09-27T15:56:24.7413508Z Generating XML reports...
2022-09-27T15:56:24.7451999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155620.xml
2022-09-27T15:56:26.8420152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:26.8421933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:26.8422554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:26.8423015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:27.1073689Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:27.1089738Z 
2022-09-27T15:56:27.1090025Z Running tests...
2022-09-27T15:56:27.1090472Z ----------------------------------------------------------------------
2022-09-27T15:56:28.7707138Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:28.7905818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11043
2022-09-27T15:56:28.7913578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11044
2022-09-27T15:56:28.7920598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11045
2022-09-27T15:56:28.7929033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11046
2022-09-27T15:56:30.4886459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:30.4887006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:30.4887594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:30.4888056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:30.4895419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:30.4896035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.4896647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:30.4897127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.5010267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:30.5032700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:30.5033642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:30.5034114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.5083825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:30.5084337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:30.5095177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:30.5095672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.7792885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:30.7896264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:30.7897357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:30.7897861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:33.8055090Z ok (6.696s)
2022-09-27T15:56:33.8055338Z 
2022-09-27T15:56:33.8055782Z ----------------------------------------------------------------------
2022-09-27T15:56:33.8056148Z Ran 1 test in 6.696s
2022-09-27T15:56:33.8056324Z 
2022-09-27T15:56:33.8056439Z OK
2022-09-27T15:56:33.8056590Z 
2022-09-27T15:56:33.8056737Z Generating XML reports...
2022-09-27T15:56:33.8094565Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155627.xml
2022-09-27T15:56:35.8808117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:35.8809148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:35.8810815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:35.8811805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:36.1411814Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:36.1427888Z 
2022-09-27T15:56:36.1428377Z Running tests...
2022-09-27T15:56:36.1428877Z ----------------------------------------------------------------------
2022-09-27T15:56:37.7551773Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:37.7743655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11254
2022-09-27T15:56:37.7751344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11255
2022-09-27T15:56:37.7757778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11256
2022-09-27T15:56:37.7764747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11257
2022-09-27T15:56:39.4567424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.4567952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.4570923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.4571545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.4576635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.4577391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:39.4581887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.4582693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:39.4826029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.4826823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.4837331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.4837914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:39.5426504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.5427082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.5441130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.5441980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:39.7315143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:39.7381762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:39.7423984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:39.8076784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:40.4838500Z ok (4.341s)
2022-09-27T15:56:40.4838739Z 
2022-09-27T15:56:40.4839206Z ----------------------------------------------------------------------
2022-09-27T15:56:40.4839571Z Ran 1 test in 4.341s
2022-09-27T15:56:40.4839744Z 
2022-09-27T15:56:40.4839837Z OK
2022-09-27T15:56:40.4839935Z 
2022-09-27T15:56:40.4840045Z Generating XML reports...
2022-09-27T15:56:40.4880708Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155636.xml
2022-09-27T15:56:42.5786644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:42.5787554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:42.5788484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:42.5788994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:42.8482594Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:42.8498571Z 
2022-09-27T15:56:42.8498805Z Running tests...
2022-09-27T15:56:42.8499474Z ----------------------------------------------------------------------
2022-09-27T15:56:44.4827372Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:44.5018535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11437
2022-09-27T15:56:44.5026386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11438
2022-09-27T15:56:44.5032737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11439
2022-09-27T15:56:44.5039318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11440
2022-09-27T15:56:46.1773044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.1774100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.1781558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.1782556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.1784122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.1785067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.1792519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.1793471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.2045274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.2046573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.2047699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.2048614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.2057176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.2058054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.2059151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.2061036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.4604028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:46.4723538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:46.4762406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:46.4880321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:47.1112921Z ok (4.261s)
2022-09-27T15:56:47.1113148Z 
2022-09-27T15:56:47.1113595Z ----------------------------------------------------------------------
2022-09-27T15:56:47.1113949Z Ran 1 test in 4.261s
2022-09-27T15:56:47.1114115Z 
2022-09-27T15:56:47.1114175Z OK
2022-09-27T15:56:47.1114318Z 
2022-09-27T15:56:47.1114437Z Generating XML reports...
2022-09-27T15:56:47.1153921Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155642.xml
2022-09-27T15:56:49.2353110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:49.2353646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:49.2355588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:49.2356788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:49.5027411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:49.5042847Z 
2022-09-27T15:56:49.5043114Z Running tests...
2022-09-27T15:56:49.5043545Z ----------------------------------------------------------------------
2022-09-27T15:56:51.1386881Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:51.1577388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11620
2022-09-27T15:56:51.1585245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11621
2022-09-27T15:56:51.1591601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11622
2022-09-27T15:56:51.1598269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11623
2022-09-27T15:56:52.8367456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.8368037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.8377356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.8377829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.8715124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.8715608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.8726569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.8727342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.9141406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.9141922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.9153101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.9153573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.9200670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.9201137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.9215084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.9215560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:53.1166912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:56:53.1271231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:56:53.1805511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:53.1864567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:55.6715169Z ok (6.167s)
2022-09-27T15:56:55.6715442Z 
2022-09-27T15:56:55.6715870Z ----------------------------------------------------------------------
2022-09-27T15:56:55.6716243Z Ran 1 test in 6.167s
2022-09-27T15:56:55.6716404Z 
2022-09-27T15:56:55.6716507Z OK
2022-09-27T15:56:55.6716624Z 
2022-09-27T15:56:55.6716764Z Generating XML reports...
2022-09-27T15:56:55.6756588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155649.xml
2022-09-27T15:56:57.8080553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:57.8081118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:57.8082644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:57.8083301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:58.0649046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:56:58.0663228Z 
2022-09-27T15:56:58.0663613Z Running tests...
2022-09-27T15:56:58.0664165Z ----------------------------------------------------------------------
2022-09-27T15:56:59.6891789Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:59.7082430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11807
2022-09-27T15:56:59.7090054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11808
2022-09-27T15:56:59.7096411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11809
2022-09-27T15:56:59.7102750Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11810
2022-09-27T15:57:01.3852914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.3853466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.3862387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.3862895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.3903202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.3903944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.3916169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.3916994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.4177319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.4177809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.4188892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.4189470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.5004650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.5005194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.5019812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.5020332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:01.6686638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:01.6775437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:01.6783000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:01.7724226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:02.4177692Z ok (4.351s)
2022-09-27T15:57:02.4177928Z 
2022-09-27T15:57:02.4178353Z ----------------------------------------------------------------------
2022-09-27T15:57:02.4178729Z Ran 1 test in 4.351s
2022-09-27T15:57:02.4178912Z 
2022-09-27T15:57:02.4178988Z OK
2022-09-27T15:57:02.4179129Z 
2022-09-27T15:57:02.4179303Z Generating XML reports...
2022-09-27T15:57:02.4216341Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155658.xml
2022-09-27T15:57:04.5436273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:04.5436840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:04.5437460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:04.5437948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:04.8124300Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:04.8139899Z 
2022-09-27T15:57:04.8140081Z Running tests...
2022-09-27T15:57:04.8140551Z ----------------------------------------------------------------------
2022-09-27T15:57:06.4604956Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:06.4804326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11990
2022-09-27T15:57:06.4811758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11991
2022-09-27T15:57:06.4818836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11992
2022-09-27T15:57:06.4825958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11993
2022-09-27T15:57:08.1644374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:08.1645368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:08.1653269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:08.1654204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.1717886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:08.1718913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:08.1721679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:08.1722560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:08.1729546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:08.1730818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.1733378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:08.1734256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.2077154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:08.2078071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:08.2089202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:08.2090440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.4543793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:08.4568347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:08.4575820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:08.4647059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:09.1899839Z ok (4.376s)
2022-09-27T15:57:09.1900093Z 
2022-09-27T15:57:09.1900551Z ----------------------------------------------------------------------
2022-09-27T15:57:09.1900912Z Ran 1 test in 4.376s
2022-09-27T15:57:09.1901068Z 
2022-09-27T15:57:09.1901145Z OK
2022-09-27T15:57:09.1901630Z 
2022-09-27T15:57:09.1901779Z Generating XML reports...
2022-09-27T15:57:09.1940208Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155704.xml
2022-09-27T15:57:11.2573295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:11.2573847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:11.2574670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:11.2575176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:11.5148972Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:11.5163285Z 
2022-09-27T15:57:11.5163778Z Running tests...
2022-09-27T15:57:11.5164298Z ----------------------------------------------------------------------
2022-09-27T15:57:13.1363721Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:13.1554908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12197
2022-09-27T15:57:13.1562105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12198
2022-09-27T15:57:13.1568604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12199
2022-09-27T15:57:13.1576269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12200
2022-09-27T15:57:14.8391416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.8391955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.8392905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.8393395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.8393977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.8394443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.8399946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.8400461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:14.8401053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.8401977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:14.8402770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.8403262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:14.8562836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.8563335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.8574759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.8575257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:15.1502525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:15.1506187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:15.1561327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:15.1605930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:18.0720728Z ok (6.555s)
2022-09-27T15:57:18.0720965Z 
2022-09-27T15:57:18.0721416Z ----------------------------------------------------------------------
2022-09-27T15:57:18.0722058Z Ran 1 test in 6.556s
2022-09-27T15:57:18.0722219Z 
2022-09-27T15:57:18.0722330Z OK
2022-09-27T15:57:18.0722525Z 
2022-09-27T15:57:18.0722641Z Generating XML reports...
2022-09-27T15:57:18.0759998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155711.xml
2022-09-27T15:57:20.1706676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:20.1707210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:20.1708023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:20.1708551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:20.4463080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:20.4478598Z 
2022-09-27T15:57:20.4478857Z Running tests...
2022-09-27T15:57:20.4479286Z ----------------------------------------------------------------------
2022-09-27T15:57:22.1009300Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:22.1198408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12408
2022-09-27T15:57:22.1206541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12409
2022-09-27T15:57:22.1213914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12410
2022-09-27T15:57:22.1222580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12411
2022-09-27T15:57:23.8129869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:23.8130483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:23.8138398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:23.8138922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:23.8230707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:23.8231189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:23.8241818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:23.8242324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:23.8247813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:23.8248293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:23.8260140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:23.8260645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:23.8310296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:23.8310770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:23.8322081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:23.8322553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:24.1070834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:24.1071406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:24.1130707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:24.1193315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:24.7298018Z ok (4.282s)
2022-09-27T15:57:24.7298473Z 
2022-09-27T15:57:24.7298876Z ----------------------------------------------------------------------
2022-09-27T15:57:24.7299249Z Ran 1 test in 4.282s
2022-09-27T15:57:24.7299437Z 
2022-09-27T15:57:24.7299522Z OK
2022-09-27T15:57:24.7299690Z 
2022-09-27T15:57:24.7299805Z Generating XML reports...
2022-09-27T15:57:24.7336920Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155720.xml
2022-09-27T15:57:26.8018513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.8019558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.8020768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.8021733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:27.0721316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:27.0738055Z 
2022-09-27T15:57:27.0738533Z Running tests...
2022-09-27T15:57:27.0739045Z ----------------------------------------------------------------------
2022-09-27T15:57:28.7083132Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:28.7283242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12591
2022-09-27T15:57:28.7291746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12592
2022-09-27T15:57:28.7300434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12593
2022-09-27T15:57:28.7307171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12594
2022-09-27T15:57:30.4139109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.4140200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.4147256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.4148249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.4164851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.4165797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.4176605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.4177588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.4699283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.4700227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.4709709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.4710681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.4860479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:30.4861356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:30.4875218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:30.4876219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:30.6740772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:30.6944367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:30.7309131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:30.7507735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:31.3381082Z ok (4.264s)
2022-09-27T15:57:31.3381368Z 
2022-09-27T15:57:31.3381805Z ----------------------------------------------------------------------
2022-09-27T15:57:31.3382144Z Ran 1 test in 4.264s
2022-09-27T15:57:31.3382318Z 
2022-09-27T15:57:31.3382422Z OK
2022-09-27T15:57:31.3382565Z 
2022-09-27T15:57:31.3382706Z Generating XML reports...
2022-09-27T15:57:31.3420164Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155727.xml
2022-09-27T15:57:33.4415596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:33.4416663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:33.4417856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:33.4418825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:33.6999035Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:33.7014523Z 
2022-09-27T15:57:33.7015009Z Running tests...
2022-09-27T15:57:33.7015503Z ----------------------------------------------------------------------
2022-09-27T15:57:35.2951730Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:35.3142191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12774
2022-09-27T15:57:35.3150103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12775
2022-09-27T15:57:35.3156228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12776
2022-09-27T15:57:35.3163120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12777
2022-09-27T15:57:36.9862627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:36.9863687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:36.9873381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:36.9874381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:36.9924620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:36.9925130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:36.9936327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:36.9936839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.0117268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.0118219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.0128803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.0130087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.0460936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.0461918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.0473055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.0474032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.2687227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:37.2801133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:37.2837098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:37.2990432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:39.7280240Z ok (6.026s)
2022-09-27T15:57:39.7280468Z 
2022-09-27T15:57:39.7280905Z ----------------------------------------------------------------------
2022-09-27T15:57:39.7281258Z Ran 1 test in 6.026s
2022-09-27T15:57:39.7281379Z 
2022-09-27T15:57:39.7281471Z OK
2022-09-27T15:57:39.7281617Z 
2022-09-27T15:57:39.7281758Z Generating XML reports...
2022-09-27T15:57:39.7318757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155733.xml
2022-09-27T15:57:41.8035359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:41.8035902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:41.8036751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:41.8037255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:42.0633200Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:42.0647655Z 
2022-09-27T15:57:42.0647962Z Running tests...
2022-09-27T15:57:43.6706861Z ----------------------------------------------------------------------
2022-09-27T15:57:43.6707741Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:43.6898427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12961
2022-09-27T15:57:43.6905373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12962
2022-09-27T15:57:43.6911999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12963
2022-09-27T15:57:43.6918453Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12964
2022-09-27T15:57:45.3859359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:45.3859884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:45.3860485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:45.3860956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:45.3868259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:45.3868735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:45.3869356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:45.3869838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:45.3880824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:45.3881274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:45.3892389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:45.3892889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:45.4036099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:45.4036550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:45.4046674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:45.4047499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:45.6646442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:45.6830294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:45.6845463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:45.6848805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:46.2999545Z ok (4.235s)
2022-09-27T15:57:46.2999784Z 
2022-09-27T15:57:46.3000262Z ----------------------------------------------------------------------
2022-09-27T15:57:46.3000593Z Ran 1 test in 4.235s
2022-09-27T15:57:46.3000767Z 
2022-09-27T15:57:46.3000871Z OK
2022-09-27T15:57:46.3001015Z 
2022-09-27T15:57:46.3001167Z Generating XML reports...
2022-09-27T15:57:46.3038905Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155742.xml
2022-09-27T15:57:48.3991831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.3992348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.3993548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.3994016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.6688595Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:48.6704855Z 
2022-09-27T15:57:48.6705373Z Running tests...
2022-09-27T15:57:48.6705852Z ----------------------------------------------------------------------
2022-09-27T15:57:50.3255318Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:50.3453488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13144
2022-09-27T15:57:50.3461122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13145
2022-09-27T15:57:50.3467888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13146
2022-09-27T15:57:50.3474723Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13147
2022-09-27T15:57:52.1068735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:52.1069280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:52.1077926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:52.1078413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.1438357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:52.1438775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:52.1449004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:52.1449498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.1599769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:52.1600230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:52.1612334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:52.1612789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.1817932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:52.1818690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:52.1828897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:52.1829358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.4198273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:52.4373517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:52.4385809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:52.4415881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:53.0548710Z ok (4.384s)
2022-09-27T15:57:53.0548946Z 
2022-09-27T15:57:53.0549407Z ----------------------------------------------------------------------
2022-09-27T15:57:53.0549763Z Ran 1 test in 4.384s
2022-09-27T15:57:53.0549928Z 
2022-09-27T15:57:53.0549976Z OK
2022-09-27T15:57:53.0550117Z 
2022-09-27T15:57:53.0550260Z Generating XML reports...
2022-09-27T15:57:53.0588267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155748.xml
2022-09-27T15:57:55.1201404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:55.1201980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:55.1206022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:55.1206999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:55.3824928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:57:55.3839256Z 
2022-09-27T15:57:55.3839597Z Running tests...
2022-09-27T15:57:55.3840037Z ----------------------------------------------------------------------
2022-09-27T15:57:56.9958892Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:57.0151603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13327
2022-09-27T15:57:57.0158496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13328
2022-09-27T15:57:57.0164840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13329
2022-09-27T15:57:57.0171405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13330
2022-09-27T15:57:58.6921633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:58.6922526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:58.6935635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:58.6936586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.7126973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:58.7127478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:58.7136212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:58.7137141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:58.7139339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:58.7140242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.7152331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:58.7153672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.7251371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:58.7252279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:58.7264665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:58.7265505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.9948093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:57:58.9976267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:58.9980718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:57:59.0082124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:00.2258449Z ok (4.842s)
2022-09-27T15:58:00.2258726Z 
2022-09-27T15:58:00.2259146Z ----------------------------------------------------------------------
2022-09-27T15:58:00.2259507Z Ran 1 test in 4.842s
2022-09-27T15:58:00.2259686Z 
2022-09-27T15:58:00.2259763Z OK
2022-09-27T15:58:00.2259936Z 
2022-09-27T15:58:00.2260057Z Generating XML reports...
2022-09-27T15:58:00.2299265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155755.xml
2022-09-27T15:58:02.3179422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:02.3179976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:02.3181261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:02.3181763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:02.5766183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:02.5780645Z 
2022-09-27T15:58:02.5780850Z Running tests...
2022-09-27T15:58:02.5781400Z ----------------------------------------------------------------------
2022-09-27T15:58:04.1776737Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:04.1967860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13534
2022-09-27T15:58:04.1975568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13535
2022-09-27T15:58:04.1983099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13536
2022-09-27T15:58:04.1988989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13537
2022-09-27T15:58:05.8794464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:05.8795039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:05.8798122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:05.8798637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:05.8804000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:05.8804544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:05.8809602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:05.8810304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:05.8822133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:05.8822964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:05.8834326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:05.8834846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:05.8991645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:05.8992118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:05.9004390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:05.9004932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:06.1642619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:06.1653864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:06.1769422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:06.1777101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:10.4139809Z ok (7.835s)
2022-09-27T15:58:10.4140119Z 
2022-09-27T15:58:10.4140550Z ----------------------------------------------------------------------
2022-09-27T15:58:10.4140877Z Ran 1 test in 7.836s
2022-09-27T15:58:10.4141030Z 
2022-09-27T15:58:10.4141139Z OK
2022-09-27T15:58:10.4141283Z 
2022-09-27T15:58:10.4141426Z Generating XML reports...
2022-09-27T15:58:10.4178683Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155802.xml
2022-09-27T15:58:12.5606392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.5606986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.5607965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.5608468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.8258739Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:12.8273027Z 
2022-09-27T15:58:12.8273346Z Running tests...
2022-09-27T15:58:12.8273790Z ----------------------------------------------------------------------
2022-09-27T15:58:14.4415644Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:14.4606551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13745
2022-09-27T15:58:14.4614390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13746
2022-09-27T15:58:14.4621367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13747
2022-09-27T15:58:14.4628179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13748
2022-09-27T15:58:16.1473973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1474516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1481981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1483052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.1483760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1484229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1495338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1496069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.1649514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1650525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1661958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1662705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.1850193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1850903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1861512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1862264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.4259477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:16.4457943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:16.4467992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:16.4469409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:17.1702120Z ok (4.342s)
2022-09-27T15:58:17.1702548Z 
2022-09-27T15:58:17.1703346Z ----------------------------------------------------------------------
2022-09-27T15:58:17.1703754Z Ran 1 test in 4.343s
2022-09-27T15:58:17.1703909Z 
2022-09-27T15:58:17.1704015Z OK
2022-09-27T15:58:17.1704160Z 
2022-09-27T15:58:17.1704301Z Generating XML reports...
2022-09-27T15:58:17.1742888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155812.xml
2022-09-27T15:58:19.2502672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:19.2503337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:19.2504993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:19.2505823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:19.5209084Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:19.5224477Z 
2022-09-27T15:58:19.5224813Z Running tests...
2022-09-27T15:58:19.5225769Z ----------------------------------------------------------------------
2022-09-27T15:58:21.1915172Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:21.2116334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13932
2022-09-27T15:58:21.2124283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13933
2022-09-27T15:58:21.2131145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13934
2022-09-27T15:58:21.2138748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13935
2022-09-27T15:58:22.9066826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:22.9067860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:22.9075301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:22.9076235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:22.9114073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:22.9114940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:22.9125892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:22.9126827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:22.9423647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:22.9424531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:22.9435892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:22.9436822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:22.9918674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:22.9919731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:22.9932774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:22.9933688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:23.1917712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:23.1958420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:23.2034610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:23.2537289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:23.9213403Z ok (4.398s)
2022-09-27T15:58:23.9213629Z 
2022-09-27T15:58:23.9214625Z ----------------------------------------------------------------------
2022-09-27T15:58:23.9215020Z Ran 1 test in 4.399s
2022-09-27T15:58:23.9215219Z 
2022-09-27T15:58:23.9216618Z OK
2022-09-27T15:58:23.9216954Z 
2022-09-27T15:58:23.9217249Z Generating XML reports...
2022-09-27T15:58:23.9254468Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155819.xml
2022-09-27T15:58:26.0552531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:26.0553070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:26.0553897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:26.0554396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:26.3281115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:26.3297482Z 
2022-09-27T15:58:26.3297942Z Running tests...
2022-09-27T15:58:26.3298461Z ----------------------------------------------------------------------
2022-09-27T15:58:27.9968714Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:28.0167556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14115
2022-09-27T15:58:28.0175017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14116
2022-09-27T15:58:28.0182465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14117
2022-09-27T15:58:28.0188953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14118
2022-09-27T15:58:29.8186533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:29.8187041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:29.8196085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.8196597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.8261243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:29.8261697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:29.8272923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.8273419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.8563565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:29.8564050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:29.8574786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.8575290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.8599692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:29.8600163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:29.8610862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:29.8611337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:30.1098012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:30.1153116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:30.1200876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:30.1202040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:32.6312280Z ok (6.301s)
2022-09-27T15:58:32.6312531Z 
2022-09-27T15:58:32.6312974Z ----------------------------------------------------------------------
2022-09-27T15:58:32.6313392Z Ran 1 test in 6.301s
2022-09-27T15:58:32.6313573Z 
2022-09-27T15:58:32.6313657Z OK
2022-09-27T15:58:32.6313794Z 
2022-09-27T15:58:32.6313938Z Generating XML reports...
2022-09-27T15:58:32.6353780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155826.xml
2022-09-27T15:58:34.7589184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:34.7589712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:34.7590814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:34.7591362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:35.0278336Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:35.0294859Z 
2022-09-27T15:58:35.0295105Z Running tests...
2022-09-27T15:58:35.0295527Z ----------------------------------------------------------------------
2022-09-27T15:58:36.6815851Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:36.7003818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14302
2022-09-27T15:58:36.7011290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14303
2022-09-27T15:58:36.7018463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14304
2022-09-27T15:58:36.7025036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14305
2022-09-27T15:58:38.3751248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.3751822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.3760097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.3760877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.3774233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.3774731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.3785895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.3786402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.3844748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.3845214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.3856458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.3856963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.4016080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.4016566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.4027953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.4028459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.6546189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:38.6573701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:38.6619138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:38.6730690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:39.3106306Z ok (4.281s)
2022-09-27T15:58:39.3106549Z 
2022-09-27T15:58:39.3107017Z ----------------------------------------------------------------------
2022-09-27T15:58:39.3107310Z Ran 1 test in 4.281s
2022-09-27T15:58:39.3107484Z 
2022-09-27T15:58:39.3107585Z OK
2022-09-27T15:58:39.3107728Z 
2022-09-27T15:58:39.3107875Z Generating XML reports...
2022-09-27T15:58:39.3146506Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155835.xml
2022-09-27T15:58:41.3787868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:41.3788465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:41.3789523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:41.3790304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:41.6519070Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:41.6534925Z 
2022-09-27T15:58:41.6535350Z Running tests...
2022-09-27T15:58:41.6535832Z ----------------------------------------------------------------------
2022-09-27T15:58:43.2885968Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:43.3076750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14485
2022-09-27T15:58:43.3084589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14486
2022-09-27T15:58:43.3091764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14487
2022-09-27T15:58:43.3099045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14488
2022-09-27T15:58:44.9885080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:44.9886441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:44.9893950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:44.9894930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:44.9898504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:44.9899387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:44.9910036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:44.9911047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:45.0210362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:45.0211290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:45.0222584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:45.0223529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:45.0567813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:45.0568794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:45.0579541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:45.0580515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:45.2633996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:45.2783278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:45.2830714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:45.3068974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:46.3183835Z ok (4.664s)
2022-09-27T15:58:46.3184211Z 
2022-09-27T15:58:46.3184654Z ----------------------------------------------------------------------
2022-09-27T15:58:46.3184999Z Ran 1 test in 4.665s
2022-09-27T15:58:46.3185295Z 
2022-09-27T15:58:46.3185472Z OK
2022-09-27T15:58:46.3185703Z 
2022-09-27T15:58:46.3188233Z Generating XML reports...
2022-09-27T15:58:46.3224936Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155841.xml
2022-09-27T15:58:48.4362907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:48.4363413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:48.4364887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:48.4365430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:48.7027508Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:48.7043210Z 
2022-09-27T15:58:48.7043642Z Running tests...
2022-09-27T15:58:48.7044141Z ----------------------------------------------------------------------
2022-09-27T15:58:50.3666401Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:50.3863812Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14692
2022-09-27T15:58:50.3871447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14693
2022-09-27T15:58:50.3878419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14694
2022-09-27T15:58:50.3884912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14695
2022-09-27T15:58:52.0722605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.0723127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.0723735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.0724197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.0731286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.0731927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.0732570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.0733048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.0762915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.0763369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.0774736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.0775202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.1002556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.1003000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.1014917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.1015415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.3538737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:58:52.3667314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:52.3696763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:58:52.3785866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:55.7020271Z ok (6.997s)
2022-09-27T15:58:55.7020590Z 
2022-09-27T15:58:55.7021019Z ----------------------------------------------------------------------
2022-09-27T15:58:55.7021340Z Ran 1 test in 6.997s
2022-09-27T15:58:55.7021484Z 
2022-09-27T15:58:55.7021591Z OK
2022-09-27T15:58:55.7021732Z 
2022-09-27T15:58:55.7021875Z Generating XML reports...
2022-09-27T15:58:55.7057891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155848.xml
2022-09-27T15:58:57.7795961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:57.7796850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:57.7797922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:57.7798418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:58.0486398Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:58:58.0502944Z 
2022-09-27T15:58:58.0503444Z Running tests...
2022-09-27T15:58:58.0504008Z ----------------------------------------------------------------------
2022-09-27T15:58:59.6884649Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:59.7074470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14903
2022-09-27T15:58:59.7082378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14904
2022-09-27T15:58:59.7088918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14905
2022-09-27T15:58:59.7095935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14906
2022-09-27T15:59:01.3798688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.3799217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.3807446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.3807897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:01.3823874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.3824334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.3827238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.3827685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.3835149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.3835648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:01.3838592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.3839065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:01.4092240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.4092734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.4103414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.4103926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:01.6702956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:01.6715156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:01.6725116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:01.6820909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:02.4171268Z ok (4.366s)
2022-09-27T15:59:02.4171542Z 
2022-09-27T15:59:02.4171976Z ----------------------------------------------------------------------
2022-09-27T15:59:02.4172342Z Ran 1 test in 4.367s
2022-09-27T15:59:02.4172515Z 
2022-09-27T15:59:02.4172618Z OK
2022-09-27T15:59:02.4172792Z 
2022-09-27T15:59:02.4172883Z Generating XML reports...
2022-09-27T15:59:02.4212279Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155858.xml
2022-09-27T15:59:04.5386598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:04.5387157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:04.5388137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:04.5388613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:04.7999246Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:04.8014248Z 
2022-09-27T15:59:04.8014722Z Running tests...
2022-09-27T15:59:04.8015248Z ----------------------------------------------------------------------
2022-09-27T15:59:06.4107591Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:06.4298663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15098
2022-09-27T15:59:06.4306245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15099
2022-09-27T15:59:06.4313051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15100
2022-09-27T15:59:06.4319649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15101
2022-09-27T15:59:08.1921793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:08.1922319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:08.1930284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:08.1930950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:08.1996069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:08.1996542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:08.2007075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:08.2007570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:08.2126875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:08.2127321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:08.2138955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:08.2428215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:08.2428873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:08.2429335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:08.2439001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:08.2439498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:08.4765543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:08.4841545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:08.4881427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:08.4983178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:09.3401283Z ok (4.538s)
2022-09-27T15:59:09.3401523Z 
2022-09-27T15:59:09.3401940Z ----------------------------------------------------------------------
2022-09-27T15:59:09.3402310Z Ran 1 test in 4.539s
2022-09-27T15:59:09.3402487Z 
2022-09-27T15:59:09.3402588Z OK
2022-09-27T15:59:09.3402729Z 
2022-09-27T15:59:09.3402868Z Generating XML reports...
2022-09-27T15:59:09.3441613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155904.xml
2022-09-27T15:59:11.4475441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:11.4475959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:11.4476797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:11.4477319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:11.7050805Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:11.7066349Z 
2022-09-27T15:59:11.7066974Z Running tests...
2022-09-27T15:59:11.7067474Z ----------------------------------------------------------------------
2022-09-27T15:59:13.3024685Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:13.3214257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15317
2022-09-27T15:59:13.3221040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15318
2022-09-27T15:59:13.3227523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15319
2022-09-27T15:59:13.3234580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15320
2022-09-27T15:59:15.0631674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.0632667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.0640105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.0641064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.0642245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.0643074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.0651439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.0652424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.0774650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.0775917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.0787658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.0788644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.0901838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:15.0902730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:15.0912893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:15.0913824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:15.3413578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:15.3467780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:15.3528987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:15.3532906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:15.9307344Z ok (4.224s)
2022-09-27T15:59:15.9307594Z 
2022-09-27T15:59:15.9308031Z ----------------------------------------------------------------------
2022-09-27T15:59:15.9308401Z Ran 1 test in 4.224s
2022-09-27T15:59:15.9308541Z 
2022-09-27T15:59:15.9308665Z OK
2022-09-27T15:59:15.9308806Z 
2022-09-27T15:59:15.9308939Z Generating XML reports...
2022-09-27T15:59:15.9347341Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155911.xml
2022-09-27T15:59:18.0262310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:18.0262870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:18.0264256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:18.0264726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:18.2915425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:18.2930968Z 
2022-09-27T15:59:18.2931409Z Running tests...
2022-09-27T15:59:18.2931873Z ----------------------------------------------------------------------
2022-09-27T15:59:19.9494399Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:19.9692177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15500
2022-09-27T15:59:19.9700268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15501
2022-09-27T15:59:19.9706838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15502
2022-09-27T15:59:19.9713655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15503
2022-09-27T15:59:21.6491241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.6492326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.6501764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.6502716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.6536472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.6536981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.6547375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.6547873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.6556483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.6556993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.6567538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.6568043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.6710936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.6711842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.6722612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.6723485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.9358675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:21.9370640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:21.9371618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:21.9454280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:24.3828841Z ok (6.089s)
2022-09-27T15:59:24.3829092Z 
2022-09-27T15:59:24.3829541Z ----------------------------------------------------------------------
2022-09-27T15:59:24.3829884Z Ran 1 test in 6.090s
2022-09-27T15:59:24.3830048Z 
2022-09-27T15:59:24.3830134Z OK
2022-09-27T15:59:24.3830235Z 
2022-09-27T15:59:24.3830371Z Generating XML reports...
2022-09-27T15:59:24.3868336Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155918.xml
2022-09-27T15:59:26.5124827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:26.5125329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:26.5126908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:26.5127406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:26.7832751Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:26.7848816Z 
2022-09-27T15:59:26.7849275Z Running tests...
2022-09-27T15:59:26.7850073Z ----------------------------------------------------------------------
2022-09-27T15:59:28.4541299Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:28.4741708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15687
2022-09-27T15:59:28.4750666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15688
2022-09-27T15:59:28.4759664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15689
2022-09-27T15:59:28.4768830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15690
2022-09-27T15:59:30.1510940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.1512014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.1513210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.1514145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.1519784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.1520733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.1522382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.1523349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.1748571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.1749494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.1759565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.1760481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.2197424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.2198308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.2209841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.2210669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.4266840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:30.4309399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:30.4428732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:30.4831820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:31.1848743Z ok (4.400s)
2022-09-27T15:59:31.1848983Z 
2022-09-27T15:59:31.1849387Z ----------------------------------------------------------------------
2022-09-27T15:59:31.1849995Z Ran 1 test in 4.400s
2022-09-27T15:59:31.1850148Z 
2022-09-27T15:59:31.1850253Z OK
2022-09-27T15:59:31.1850403Z 
2022-09-27T15:59:31.1850534Z Generating XML reports...
2022-09-27T15:59:31.1890116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155926.xml
2022-09-27T15:59:33.3041232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:33.3042130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:33.3043113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:33.3043623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:33.5717506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:33.5733298Z 
2022-09-27T15:59:33.5733618Z Running tests...
2022-09-27T15:59:33.5734066Z ----------------------------------------------------------------------
2022-09-27T15:59:35.2399953Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:35.2599579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15870
2022-09-27T15:59:35.2607406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15871
2022-09-27T15:59:35.2614662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15872
2022-09-27T15:59:35.2621527Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15873
2022-09-27T15:59:36.9460423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.9460968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.9468694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.9469200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.9477759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.9478311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.9488912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.9489464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.9598601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.9599108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.9609795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.9610516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.9713832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.9714504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.9726161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.9726718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:37.2251652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:37.2363366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:37.2364121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:37.2436945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:38.5706555Z ok (4.997s)
2022-09-27T15:59:38.5706811Z 
2022-09-27T15:59:38.5707274Z ----------------------------------------------------------------------
2022-09-27T15:59:38.5707632Z Ran 1 test in 4.997s
2022-09-27T15:59:38.5707795Z 
2022-09-27T15:59:38.5707901Z OK
2022-09-27T15:59:38.5708038Z 
2022-09-27T15:59:38.5708181Z Generating XML reports...
2022-09-27T15:59:38.5746663Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155933.xml
2022-09-27T15:59:40.6719509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:40.6720059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:40.6721591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:40.6722095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:40.9287640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:40.9302423Z 
2022-09-27T15:59:40.9302683Z Running tests...
2022-09-27T15:59:40.9303165Z ----------------------------------------------------------------------
2022-09-27T15:59:40.9309683Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-09-27T15:59:40.9310103Z 
2022-09-27T15:59:40.9310418Z ----------------------------------------------------------------------
2022-09-27T15:59:40.9310762Z Ran 1 test in 0.001s
2022-09-27T15:59:40.9310939Z 
2022-09-27T15:59:40.9311040Z OK (skipped=1)
2022-09-27T15:59:40.9311201Z 
2022-09-27T15:59:40.9311335Z Generating XML reports...
2022-09-27T15:59:40.9342860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155940.xml
2022-09-27T15:59:42.8894109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.8894636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.8896451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.8896984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:43.1568145Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:43.1584539Z 
2022-09-27T15:59:43.1584992Z Running tests...
2022-09-27T15:59:43.1585491Z ----------------------------------------------------------------------
2022-09-27T15:59:44.8118237Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:44.8310326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16110
2022-09-27T15:59:44.8317583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16111
2022-09-27T15:59:44.8324075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16112
2022-09-27T15:59:44.8331250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16113
2022-09-27T15:59:46.5056627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5057186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.5065055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.5065579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.5066913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5067372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.5078159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.5078650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.5322328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5323127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.5334368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.5334863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.5610115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.5610609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.5621406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.5621900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.7829996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:46.7967266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:46.8032708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:46.8161607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:47.4404675Z ok (4.282s)
2022-09-27T15:59:47.4404919Z 
2022-09-27T15:59:47.4405377Z ----------------------------------------------------------------------
2022-09-27T15:59:47.4405741Z Ran 1 test in 4.282s
2022-09-27T15:59:47.4405915Z 
2022-09-27T15:59:47.4405993Z OK
2022-09-27T15:59:47.4406131Z 
2022-09-27T15:59:47.4406274Z Generating XML reports...
2022-09-27T15:59:47.4445571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155943.xml
2022-09-27T15:59:49.5059431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:49.5061064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:49.5062302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:49.5063295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:49.7642065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:49.7658770Z 
2022-09-27T15:59:49.7659152Z Running tests...
2022-09-27T15:59:49.7659694Z ----------------------------------------------------------------------
2022-09-27T15:59:49.7664450Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-09-27T15:59:49.7664780Z 
2022-09-27T15:59:49.7665083Z ----------------------------------------------------------------------
2022-09-27T15:59:49.7665416Z Ran 1 test in 0.001s
2022-09-27T15:59:49.7665589Z 
2022-09-27T15:59:49.7665710Z OK (skipped=1)
2022-09-27T15:59:49.7665873Z 
2022-09-27T15:59:49.7666006Z Generating XML reports...
2022-09-27T15:59:49.7699252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155949.xml
2022-09-27T15:59:51.7027382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:51.7027927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:51.7028811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:51.7029259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:51.9793200Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T15:59:51.9809268Z 
2022-09-27T15:59:51.9809882Z Running tests...
2022-09-27T15:59:51.9810871Z ----------------------------------------------------------------------
2022-09-27T15:59:53.6249465Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:53.6544449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16326
2022-09-27T15:59:53.6551378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16327
2022-09-27T15:59:53.6558170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16328
2022-09-27T15:59:53.6564994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16329
2022-09-27T15:59:55.3354391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:55.3354936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:55.3360791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:55.3361732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:55.3370308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:55.3371105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:55.3373372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:55.3373863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:55.3411709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:55.3412181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:55.3425280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:55.3426076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:55.3559237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:55.3559719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:55.3571654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:55.3572135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:55.6137297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T15:59:55.6173273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:55.6210473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:55.6292328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T15:59:58.2679311Z ok (6.287s)
2022-09-27T15:59:58.2679550Z 
2022-09-27T15:59:58.2679976Z ----------------------------------------------------------------------
2022-09-27T15:59:58.2680386Z Ran 1 test in 6.287s
2022-09-27T15:59:58.2680562Z 
2022-09-27T15:59:58.2680639Z OK
2022-09-27T15:59:58.2680782Z 
2022-09-27T15:59:58.2680923Z Generating XML reports...
2022-09-27T15:59:58.2719362Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155951.xml
2022-09-27T16:00:00.3490964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:00.3491501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:00.3493965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:00.3494459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:00.6071333Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:00.6085984Z 
2022-09-27T16:00:00.6086485Z Running tests...
2022-09-27T16:00:00.6086962Z ----------------------------------------------------------------------
2022-09-27T16:00:02.2307797Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:02.2504694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16693
2022-09-27T16:00:02.2512099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16694
2022-09-27T16:00:02.2518582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16695
2022-09-27T16:00:02.2525459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16696
2022-09-27T16:00:03.9412782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.9413385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.9416212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.9416700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.9421849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.9422357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.9427701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.9428173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.9530649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.9531406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.9542709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.9543192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:03.9673357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:03.9673851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:03.9684772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:03.9685248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:04.2202199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:00:04.2232374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:04.2233258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:04.2379824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:00:04.8598159Z ok (4.251s)
2022-09-27T16:00:04.8598442Z 
2022-09-27T16:00:04.8598862Z ----------------------------------------------------------------------
2022-09-27T16:00:04.8599222Z Ran 1 test in 4.251s
2022-09-27T16:00:04.8599380Z 
2022-09-27T16:00:04.8599492Z OK
2022-09-27T16:00:04.8599618Z 
2022-09-27T16:00:04.8599759Z Generating XML reports...
2022-09-27T16:00:04.8639398Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927160000.xml
2022-09-27T16:00:06.9848316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:06.9848862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:06.9850792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:06.9851300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:07.2410019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:07.2425486Z 
2022-09-27T16:00:07.2425795Z Running tests...
2022-09-27T16:00:07.2426249Z ----------------------------------------------------------------------
2022-09-27T16:00:07.2498864Z   test_forward_backward (__main__.ReducerTest) ... ok (0.007s)
2022-09-27T16:00:07.2544397Z 
2022-09-27T16:00:07.2544903Z ----------------------------------------------------------------------
2022-09-27T16:00:07.2545289Z Ran 1 test in 0.012s
2022-09-27T16:00:07.2545461Z 
2022-09-27T16:00:07.2545563Z OK
2022-09-27T16:00:07.2545686Z 
2022-09-27T16:00:07.2545827Z Generating XML reports...
2022-09-27T16:00:07.2579452Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160007.xml
2022-09-27T16:00:09.1595753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:09.1596314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:09.1598243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:09.1598738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:09.4191390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:09.4206695Z 
2022-09-27T16:00:09.4207001Z Running tests...
2022-09-27T16:00:09.4207435Z ----------------------------------------------------------------------
2022-09-27T16:00:09.4299876Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:00:09.4318307Z ok (0.011s)
2022-09-27T16:00:09.4328885Z 
2022-09-27T16:00:09.4329768Z ----------------------------------------------------------------------
2022-09-27T16:00:09.4330380Z Ran 1 test in 0.012s
2022-09-27T16:00:09.4330515Z 
2022-09-27T16:00:09.4330583Z OK
2022-09-27T16:00:09.4330726Z 
2022-09-27T16:00:09.4330859Z Generating XML reports...
2022-09-27T16:00:09.4363265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160009.xml
2022-09-27T16:00:11.3433644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:11.3434202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:11.3435282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:11.3435756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:11.6153297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:11.6168333Z 
2022-09-27T16:00:11.6168484Z Running tests...
2022-09-27T16:00:11.6169429Z ----------------------------------------------------------------------
2022-09-27T16:00:11.6248983Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.008s)
2022-09-27T16:00:11.6290438Z 
2022-09-27T16:00:11.6291124Z ----------------------------------------------------------------------
2022-09-27T16:00:11.6291865Z Ran 1 test in 0.012s
2022-09-27T16:00:11.6292153Z 
2022-09-27T16:00:11.6292263Z OK
2022-09-27T16:00:11.6292405Z 
2022-09-27T16:00:11.6292539Z Generating XML reports...
2022-09-27T16:00:11.6326310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160011.xml
2022-09-27T16:00:13.5432774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:13.5433816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:13.5435036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:13.5435902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:13.8221022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:13.8237866Z 
2022-09-27T16:00:13.8238366Z Running tests...
2022-09-27T16:00:13.8238883Z ----------------------------------------------------------------------
2022-09-27T16:00:13.8282422Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-09-27T16:00:13.8358526Z 
2022-09-27T16:00:13.8359025Z ----------------------------------------------------------------------
2022-09-27T16:00:13.8359409Z Ran 1 test in 0.012s
2022-09-27T16:00:13.8359573Z 
2022-09-27T16:00:13.8359656Z OK
2022-09-27T16:00:13.8359803Z 
2022-09-27T16:00:13.8359927Z Generating XML reports...
2022-09-27T16:00:13.8394822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160013.xml
2022-09-27T16:00:15.8210060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:15.8211140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:15.8212316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:15.8213741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:16.0914902Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:16.0931833Z 
2022-09-27T16:00:16.0932313Z Running tests...
2022-09-27T16:00:16.0932811Z ----------------------------------------------------------------------
2022-09-27T16:00:16.1001963Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-09-27T16:00:16.1051071Z 
2022-09-27T16:00:16.1051604Z ----------------------------------------------------------------------
2022-09-27T16:00:16.1051947Z Ran 1 test in 0.012s
2022-09-27T16:00:16.1052129Z 
2022-09-27T16:00:16.1052210Z OK
2022-09-27T16:00:16.1052346Z 
2022-09-27T16:00:16.1052473Z Generating XML reports...
2022-09-27T16:00:16.1085445Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160016.xml
2022-09-27T16:00:18.0655530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:18.0656579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:18.0657749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:18.0658713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:18.3330770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:18.3347128Z 
2022-09-27T16:00:18.3347567Z Running tests...
2022-09-27T16:00:18.3348082Z ----------------------------------------------------------------------
2022-09-27T16:00:18.3385494Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-09-27T16:00:18.3464605Z 
2022-09-27T16:00:18.3465152Z ----------------------------------------------------------------------
2022-09-27T16:00:18.3465463Z Ran 1 test in 0.012s
2022-09-27T16:00:18.3465696Z 
2022-09-27T16:00:18.3465794Z OK
2022-09-27T16:00:18.3465960Z 
2022-09-27T16:00:18.3466070Z Generating XML reports...
2022-09-27T16:00:18.3499550Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160018.xml
2022-09-27T16:00:20.2866371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:20.2866935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:20.2868183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:20.2868640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:20.5576548Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:20.5591899Z 
2022-09-27T16:00:20.5592051Z Running tests...
2022-09-27T16:00:20.5592522Z ----------------------------------------------------------------------
2022-09-27T16:00:22.2580993Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:22.2769924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:22.2771344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:00:22.2869656Z ok (1.728s)
2022-09-27T16:00:22.2871030Z 
2022-09-27T16:00:22.2871519Z ----------------------------------------------------------------------
2022-09-27T16:00:22.2871943Z Ran 1 test in 1.728s
2022-09-27T16:00:22.2872120Z 
2022-09-27T16:00:22.2872216Z OK
2022-09-27T16:00:22.2872366Z 
2022-09-27T16:00:22.2872467Z Generating XML reports...
2022-09-27T16:00:22.2907283Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220927160020.xml
2022-09-27T16:00:24.3303950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:24.3305021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:24.3306220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:24.3307166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:24.6028794Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-09-27T16:00:24.6045412Z 
2022-09-27T16:00:24.6045859Z Running tests...
2022-09-27T16:00:24.6046377Z ----------------------------------------------------------------------
2022-09-27T16:00:26.2426558Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:26.2590738Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.654s)
2022-09-27T16:00:26.2591293Z 
2022-09-27T16:00:26.2591634Z ----------------------------------------------------------------------
2022-09-27T16:00:26.2591988Z Ran 1 test in 1.655s
2022-09-27T16:00:26.2592158Z 
2022-09-27T16:00:26.2592243Z OK (skipped=1)
2022-09-27T16:00:26.2592428Z 
2022-09-27T16:00:26.2595226Z Generating XML reports...
2022-09-27T16:00:26.2627161Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220927160024.xml
2022-09-27T16:00:26.9770805Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-09-27 16:00:26.976499]
2022-09-27T16:00:26.9771617Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:00:26.976602]
2022-09-27T16:00:30.6051318Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-09-27T16:00:30.6075958Z 
2022-09-27T16:00:30.6076356Z Running tests...
2022-09-27T16:00:30.6076859Z ----------------------------------------------------------------------
2022-09-27T16:00:30.6247109Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17216
2022-09-27T16:00:30.6254612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17217
2022-09-27T16:00:32.3196072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:32.3196615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:32.3197511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:32.3198004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:32.3609171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:32.3609876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:32.3612798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:32.3613290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:34.3165319Z dist init r=1, world=2
2022-09-27T16:00:34.3170965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:34.3396484Z dist init r=0, world=2
2022-09-27T16:00:34.3401912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:34.3402985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:34.3476524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:35.4874978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:35.4875552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:36.7392117Z ok (6.131s)
2022-09-27T16:00:36.7413612Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17302
2022-09-27T16:00:36.7420365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17303
2022-09-27T16:00:38.4361378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:38.4361893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:38.4362577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:38.4363000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:38.4565748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:38.4566269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:38.4568228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:38.4568690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:40.4600890Z dist init r=0, world=2
2022-09-27T16:00:40.4607074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:40.4625212Z dist init r=1, world=2
2022-09-27T16:00:40.4630790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:40.4631923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:40.4709749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:41.6007899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:41.6008467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:42.2692338Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:42.2693710Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:42.2694986Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:43.2559954Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:43.2560673Z ok (6.517s)
2022-09-27T16:00:43.2581296Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17388
2022-09-27T16:00:43.2587804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17389
2022-09-27T16:00:44.8809119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:44.8810459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:44.8811645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:44.8812589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:44.9566186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:44.9567041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:44.9568046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:44.9568866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:46.8359577Z dist init r=1, world=2
2022-09-27T16:00:46.8364738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:46.8872236Z dist init r=0, world=2
2022-09-27T16:00:46.8876859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:46.8877704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:46.8975186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:48.0165189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:48.0166151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:48.6669443Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:48.6672609Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:48.6675040Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:48.6677686Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:49.5751155Z ok (6.319s)
2022-09-27T16:00:49.5771078Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17474
2022-09-27T16:00:49.5778160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17475
2022-09-27T16:00:51.2792187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:51.2793003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:51.2793620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:51.2794106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:51.3184616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:51.3185106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:51.3187693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:51.3188214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:53.2481331Z dist init r=1, world=2
2022-09-27T16:00:53.2485971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:53.2915471Z dist init r=0, world=2
2022-09-27T16:00:53.2920425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:53.2921573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:53.2994359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:54.4316245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:54.4317180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:55.0932169Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:55.0933537Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:55.0935130Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:55.0936389Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:00:55.9941018Z ok (6.419s)
2022-09-27T16:00:55.9961206Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17560
2022-09-27T16:00:55.9967801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17561
2022-09-27T16:00:57.7118633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.7119149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.7120032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.7120524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:57.7477616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.7478326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.7481457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.7481970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:59.7273771Z dist init r=0, world=2
2022-09-27T16:00:59.7274325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:59.7518538Z dist init r=1, world=2
2022-09-27T16:00:59.7524155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:59.7524997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:59.7573666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:00.9016524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:00.9017076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:01.5595127Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:01.5596495Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:01.5597767Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:01.5599271Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:02.5114118Z ok (6.517s)
2022-09-27T16:01:02.5134482Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17646
2022-09-27T16:01:02.5140645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17647
2022-09-27T16:01:04.2278993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:04.2279504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:04.2280052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:04.2280542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:04.2772386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:04.2772865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:04.2775760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:04.2776260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:06.1937579Z dist init r=1, world=2
2022-09-27T16:01:06.1942435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:06.2387678Z dist init r=0, world=2
2022-09-27T16:01:06.2392879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:06.2393756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:06.2451102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:07.3778686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:07.3779290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:08.1148178Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:08.1149529Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:08.1150799Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:08.1152038Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:09.0290359Z ok (6.517s)
2022-09-27T16:01:09.0309980Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17732
2022-09-27T16:01:09.0316111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17733
2022-09-27T16:01:10.7440382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:10.7440929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:10.7441538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:10.7442008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:10.7513694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:10.7514180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:10.7516776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:10.7517243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:12.7369207Z dist init r=1, world=2
2022-09-27T16:01:12.7369825Z dist init r=0, world=2
2022-09-27T16:01:12.7373429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:12.7373941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:12.7374788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:12.7375763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:13.8629107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:13.8629871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:14.5257514Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:14.5258862Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:14.5260116Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:14.5261370Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:15.4480340Z ok (6.419s)
2022-09-27T16:01:15.4500819Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17818
2022-09-27T16:01:15.4506932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17819
2022-09-27T16:01:17.1556497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:17.1557319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:17.1557935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:17.1558406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:17.1634999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:17.1635479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:17.1638626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:17.1639082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:19.1821550Z dist init r=0, world=2
2022-09-27T16:01:19.1826865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:19.1915356Z dist init r=1, world=2
2022-09-27T16:01:19.1920975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:19.1922147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:19.1929848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:20.3360595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:20.3361174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:21.0358590Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:21.0359987Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:21.0361237Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:21.0362485Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:22.0644077Z ok (6.616s)
2022-09-27T16:01:22.0665654Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17904
2022-09-27T16:01:22.0672308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17905
2022-09-27T16:01:23.7723647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:23.7724181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:23.7724813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:23.7725326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:23.7820400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:23.7821173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:23.7824019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:23.7824516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:25.7561675Z dist init r=0, world=2
2022-09-27T16:01:25.7566770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:25.7831477Z dist init r=1, world=2
2022-09-27T16:01:25.7836994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:25.7838220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:25.7872199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:26.9391379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:26.9391936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:27.6083037Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:27.6085035Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:27.6086678Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:27.6088205Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:28.5808701Z ok (6.516s)
2022-09-27T16:01:28.5830366Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17990
2022-09-27T16:01:28.5837123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17991
2022-09-27T16:01:30.3078343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:30.3079361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:30.3080528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:30.3081500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:30.3422908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:30.3423391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:30.3426285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:30.3426784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:32.2972604Z dist init r=1, world=2
2022-09-27T16:01:32.2978125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:32.3224607Z dist init r=0, world=2
2022-09-27T16:01:32.3230079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:32.3231109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:32.3283793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:33.4582945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:33.4583507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:34.1129147Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:34.1131245Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:34.1132819Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:34.1134605Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:34.9976273Z ok (6.417s)
2022-09-27T16:01:34.9997429Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18076
2022-09-27T16:01:35.0003717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18077
2022-09-27T16:01:36.6813574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:36.6814095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:36.6814735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:36.6815225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:36.7111630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:36.7112094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:36.7114874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:36.7115373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:38.6668928Z dist init r=1, world=2
2022-09-27T16:01:38.6674211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:38.6937779Z dist init r=0, world=2
2022-09-27T16:01:38.6943646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:38.6944683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:38.6979434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:39.8292669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:39.8293218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:40.4731069Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:40.4732614Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:40.4733883Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:40.4735151Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:41.4139659Z ok (6.416s)
2022-09-27T16:01:41.4159842Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18162
2022-09-27T16:01:41.4165776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18163
2022-09-27T16:01:43.1501789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:43.1502345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:43.1503185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:43.1503668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:43.1765459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:43.1765936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:43.1768878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:43.1769340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:45.1840396Z dist init r=1, world=2
2022-09-27T16:01:45.1845245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:45.1902396Z dist init r=0, world=2
2022-09-27T16:01:45.1908276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:45.1909691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:45.1948262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:46.3251098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:46.3251685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:46.9906614Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:46.9908317Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:46.9909602Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:46.9910848Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:47.9303561Z ok (6.516s)
2022-09-27T16:01:47.9324598Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18248
2022-09-27T16:01:47.9330899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18249
2022-09-27T16:01:49.6289261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:49.6290819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:49.6292071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:49.6293032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:49.7353143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:49.7354151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:49.7355362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:49.7356288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:51.5865858Z dist init r=0, world=2
2022-09-27T16:01:51.5870631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:51.6407768Z dist init r=1, world=2
2022-09-27T16:01:51.6413117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:51.6413947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:51.6482811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:52.7697432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:52.7698332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:53.4133051Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:53.4135796Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:53.4138658Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:53.4141295Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:54.3470358Z ok (6.417s)
2022-09-27T16:01:54.3491589Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18334
2022-09-27T16:01:54.3498495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18335
2022-09-27T16:01:56.0658576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:56.0659165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:56.0677195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:56.0677907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:56.0992883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:56.0993936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:56.0996624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:56.0997431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:58.0359230Z dist init r=1, world=2
2022-09-27T16:01:58.0364250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:58.0679040Z dist init r=0, world=2
2022-09-27T16:01:58.0683537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:58.0685005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:58.0772233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:59.2132523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:59.2133100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:59.9378180Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:59.9379494Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:59.9380772Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:01:59.9382232Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:00.9638350Z ok (6.617s)
2022-09-27T16:02:00.9660463Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18420
2022-09-27T16:02:00.9668116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18421
2022-09-27T16:02:02.7323556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:02.7324070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:02.7324680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:02.7325183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:02.7350377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:02.7350829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:02.7353705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:02.7354200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:04.7383154Z dist init r=0, world=2
2022-09-27T16:02:04.7387975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:04.7430547Z dist init r=1, world=2
2022-09-27T16:02:04.7436486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:04.7437651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:04.7490774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:05.8988875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:05.8989409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:06.5815709Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:06.5817468Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:06.5819046Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:06.5820547Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:07.5810344Z ok (6.617s)
2022-09-27T16:02:07.5832336Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18506
2022-09-27T16:02:07.5838329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18507
2022-09-27T16:02:09.3171132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:09.3171692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:09.3172309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:09.3172785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:09.3267113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:09.3267596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:09.3270289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:09.3270777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:11.3205500Z dist init r=1, world=2
2022-09-27T16:02:11.3212275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:11.3354810Z dist init r=0, world=2
2022-09-27T16:02:11.3360186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:11.3361042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:11.3415423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:12.4629053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:12.4629638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:13.1679919Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:13.1681299Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:13.1682580Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:13.1683841Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:14.0978815Z ok (6.517s)
2022-09-27T16:02:14.1000339Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18592
2022-09-27T16:02:14.1006874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18593
2022-09-27T16:02:15.7866353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:15.7866882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:15.7867756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:15.7868258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:15.7944534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:15.7945024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:15.7947690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:15.7948193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.7471671Z dist init r=0, world=2
2022-09-27T16:02:17.7476195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:17.7541324Z dist init r=1, world=2
2022-09-27T16:02:17.7546997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:17.7548014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:17.7578997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:18.8949129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:18.8949704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:19.5665036Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:19.5666462Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:19.5667729Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:19.5668980Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:20.5145472Z ok (6.417s)
2022-09-27T16:02:20.5166354Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18678
2022-09-27T16:02:20.5173171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18679
2022-09-27T16:02:22.1970930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:22.1971484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:22.1972110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:22.1972581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:22.2235407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:22.2236179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:22.2238833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:22.2239313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:24.1761214Z dist init r=1, world=2
2022-09-27T16:02:24.1765071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:24.2061331Z dist init r=0, world=2
2022-09-27T16:02:24.2067376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:24.2068517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:24.2070782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:25.3481546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:25.3482157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:25.9140530Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:25.9142178Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:26.0814891Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:26.0816182Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:26.0817439Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:26.0818681Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:26.8309462Z ok (6.316s)
2022-09-27T16:02:26.8332005Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18764
2022-09-27T16:02:26.8339518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18765
2022-09-27T16:02:28.5210875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:28.5211411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:28.5212018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:28.5212490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:28.5630489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:28.5630970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:28.5634090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:28.5634562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:30.5055877Z dist init r=1, world=2
2022-09-27T16:02:30.5060831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:30.5233907Z dist init r=0, world=2
2022-09-27T16:02:30.5239772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:30.5240666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:30.5264993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:31.6669444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:31.6669991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:32.4077064Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:32.4078603Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:32.4079878Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:32.4081106Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:33.1476206Z ok (6.317s)
2022-09-27T16:02:33.1499646Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18850
2022-09-27T16:02:33.1506743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18851
2022-09-27T16:02:34.8550852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:34.8551416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:34.8552032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:34.8552489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:34.9185105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:34.9185776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:34.9186994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:34.9187854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.8218940Z dist init r=0, world=2
2022-09-27T16:02:36.8224328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:36.8525630Z dist init r=1, world=2
2022-09-27T16:02:36.8530898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:36.8531749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:36.8632714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:38.0229981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:38.0230588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:38.6063542Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:38.6064932Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:38.7833558Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:38.7834888Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:38.7836150Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:38.7837398Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:39.5648382Z ok (6.417s)
2022-09-27T16:02:39.5670512Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18936
2022-09-27T16:02:39.5677145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18937
2022-09-27T16:02:41.2583058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:41.2583936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:41.2584922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:41.2585402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:41.2870001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:41.2870477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:41.2873599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:41.2874055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:43.2293138Z dist init r=0, world=2
2022-09-27T16:02:43.2297925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:43.2547871Z dist init r=1, world=2
2022-09-27T16:02:43.2554525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:43.2555606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:43.2604020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:44.4243250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:44.4243861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:45.1625454Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:45.1626805Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:45.1628390Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:45.1629682Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:45.8813433Z ok (6.316s)
2022-09-27T16:02:45.8834678Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19022
2022-09-27T16:02:45.8841197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19023
2022-09-27T16:02:47.5781181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:47.5781730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:47.5782545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:47.5783054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:47.6410192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:47.6410941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:47.6411790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:47.6412296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:49.5183571Z dist init r=1, world=2
2022-09-27T16:02:49.5188636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:49.6017393Z dist init r=0, world=2
2022-09-27T16:02:49.6022287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:49.6023436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:49.6101733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:50.7509454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:50.7510065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:51.4065619Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:51.4066963Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:51.4068221Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:51.4069732Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:52.4985237Z ok (6.617s)
2022-09-27T16:02:52.5006026Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19108
2022-09-27T16:02:52.5013185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19109
2022-09-27T16:02:54.2387367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:54.2387928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:54.2389090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:54.2389833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:54.3267129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:54.3268177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:54.3269131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:54.3269630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:56.1590902Z dist init r=1, world=2
2022-09-27T16:02:56.1595526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:56.2282328Z dist init r=0, world=2
2022-09-27T16:02:56.2287273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:56.2288414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:56.2306484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:57.3536131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:57.3536717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:58.1437832Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:58.1439162Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:58.1440425Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:58.1441674Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:02:58.9148248Z ok (6.416s)
2022-09-27T16:02:58.9169673Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19194
2022-09-27T16:02:58.9176420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19195
2022-09-27T16:03:00.5766446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:00.5767211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:00.5767764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:00.5768286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:00.6249389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:00.6250452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:00.6252077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:00.6252598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:02.5617876Z dist init r=1, world=2
2022-09-27T16:03:02.5622182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:02.5944942Z dist init r=0, world=2
2022-09-27T16:03:02.5950848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:02.5951704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:02.6029169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:03.7415530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:03.7416080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:04.3812342Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:04.3813970Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:04.3815236Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:04.3816474Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:05.4312548Z ok (6.516s)
2022-09-27T16:03:05.4333610Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19280
2022-09-27T16:03:05.4340320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19281
2022-09-27T16:03:07.1062369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:07.1063203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:07.1064139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:07.1064794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:07.1457088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:07.1457647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:07.1460549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:07.1461259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:09.0818061Z dist init r=0, world=2
2022-09-27T16:03:09.0823409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:09.1156495Z dist init r=1, world=2
2022-09-27T16:03:09.1161834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:09.1163276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:09.1230853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:10.2894916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:10.2895452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:11.0885063Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:11.0887782Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:11.0891330Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:11.0893748Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:11.8475268Z ok (6.416s)
2022-09-27T16:03:11.8496626Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19366
2022-09-27T16:03:11.8502858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19367
2022-09-27T16:03:13.5425972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:13.5426510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:13.5427132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:13.5427602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:13.5661776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:13.5662273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:13.5665540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:13.5666036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:15.5342775Z dist init r=0, world=2
2022-09-27T16:03:15.5347451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:15.5539421Z dist init r=1, world=2
2022-09-27T16:03:15.5545591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:15.5546447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:15.5551568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:16.7320951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:16.7321561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:17.4321844Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:17.4323211Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:17.4324467Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:17.4325717Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:18.3637323Z ok (6.516s)
2022-09-27T16:03:18.3660906Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19452
2022-09-27T16:03:18.3668614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19453
2022-09-27T16:03:20.0646513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.0647076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.0647688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.0648192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:20.0716827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.0717277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.0719933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.0720426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:22.0335673Z dist init r=1, world=2
2022-09-27T16:03:22.0340323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:22.0391869Z dist init r=0, world=2
2022-09-27T16:03:22.0397547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:22.0398421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:22.0443096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:23.1647114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:23.1647714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:23.8305035Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:23.8306426Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:23.8307660Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:23.8308912Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:24.7803032Z ok (6.416s)
2022-09-27T16:03:24.7823559Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19538
2022-09-27T16:03:24.7829795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19539
2022-09-27T16:03:26.5247888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:26.5248553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:26.5249248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:26.5249969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:26.5258084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:26.5258576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:26.5261693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:26.5262202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:28.4725218Z dist init r=0, world=2
2022-09-27T16:03:28.4731034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:28.5042349Z dist init r=1, world=2
2022-09-27T16:03:28.5047722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:28.5048965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:28.5137059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:29.6562454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:29.6563033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:30.3378250Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:30.3379618Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:30.3380906Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:30.3382168Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:31.1963719Z ok (6.416s)
2022-09-27T16:03:31.1984221Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19624
2022-09-27T16:03:31.1990191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19625
2022-09-27T16:03:32.8819694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:32.8820238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:32.8820852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:32.8821633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:32.9185895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:32.9186375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:32.9189240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:32.9189739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:34.8366918Z dist init r=0, world=2
2022-09-27T16:03:34.8371962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:34.8674471Z dist init r=1, world=2
2022-09-27T16:03:34.8680259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:34.8681125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:34.8779311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:36.0252332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:36.0252903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:36.6935361Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:36.6936720Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:36.6937996Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:36.6939255Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:37.6126997Z ok (6.416s)
2022-09-27T16:03:37.6147035Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19710
2022-09-27T16:03:37.6153376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19711
2022-09-27T16:03:39.3102011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:39.3102545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:39.3103128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:39.3103595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:39.3866650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:39.3867139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:39.3868043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:39.3868499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:41.2708134Z dist init r=0, world=2
2022-09-27T16:03:41.2711872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:41.3288319Z dist init r=1, world=2
2022-09-27T16:03:41.3294089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:41.3294891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:41.3321422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:42.4855548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:42.4856169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:43.2136980Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:43.2138321Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:43.2139865Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:43.2141142Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:44.2295939Z ok (6.617s)
2022-09-27T16:03:44.2315940Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19796
2022-09-27T16:03:44.2322262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19797
2022-09-27T16:03:45.9456147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:45.9456741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:45.9457834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:45.9458329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:45.9650182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:45.9650960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:45.9654347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:45.9654856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:47.9566222Z dist init r=1, world=2
2022-09-27T16:03:47.9571438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:47.9764241Z dist init r=0, world=2
2022-09-27T16:03:47.9770203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:47.9771276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:47.9775534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:49.1128853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:49.1129415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:49.8393593Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:49.8394938Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:49.8396199Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:49.8397732Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:50.7462896Z ok (6.517s)
2022-09-27T16:03:50.7484113Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19882
2022-09-27T16:03:50.7490229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19883
2022-09-27T16:03:52.4006561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:52.4007088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:52.4007921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:52.4008447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:52.4490407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:52.4490936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:52.4493613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:52.4494112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:54.3401807Z dist init r=0, world=2
2022-09-27T16:03:54.3407022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:54.3936395Z dist init r=1, world=2
2022-09-27T16:03:54.3941191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:54.3942459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:54.4017002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:55.5565855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:55.5566415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:56.2847831Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:56.2849203Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:56.2850748Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:56.2852117Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:03:57.2627325Z ok (6.516s)
2022-09-27T16:03:57.2648229Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19968
2022-09-27T16:03:57.2654873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19969
2022-09-27T16:03:58.9691960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:58.9692515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:58.9693314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:58.9693812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:58.9909645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:58.9910124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:58.9912796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:58.9913320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:00.9788181Z dist init r=0, world=2
2022-09-27T16:04:00.9792753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:00.9801665Z dist init r=1, world=2
2022-09-27T16:04:00.9807117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:00.9808357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:00.9895923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:02.1359052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:02.1359562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:02.8229622Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:02.8231316Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:02.8232568Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:02.8233810Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:03.7793367Z ok (6.516s)
2022-09-27T16:04:03.7814150Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20054
2022-09-27T16:04:03.7820266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20055
2022-09-27T16:04:05.4665295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:05.4665862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:05.4667039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:05.4667533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:05.4766424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:05.4767127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:05.4769643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:05.4770558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:07.4326309Z dist init r=1, world=2
2022-09-27T16:04:07.4331514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:07.4982154Z dist init r=0, world=2
2022-09-27T16:04:07.4985841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:07.4986943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:07.5042636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:08.6404904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:08.6405470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:09.3905846Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:09.3907214Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:09.3908823Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:09.3910048Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:10.1961317Z ok (6.417s)
2022-09-27T16:04:10.1981522Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20140
2022-09-27T16:04:10.1987575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20141
2022-09-27T16:04:11.8697431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:11.8698282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:11.8699465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:11.8700237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:11.9104063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:11.9104577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:11.9107598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:11.9108069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:13.8375078Z dist init r=1, world=2
2022-09-27T16:04:13.8378629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:13.8435338Z dist init r=0, world=2
2022-09-27T16:04:13.8441856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:13.8442991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:13.8481362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:14.9883005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:14.9883682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:15.7207097Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:15.7208462Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:15.7209985Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:15.7211226Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:16.5125523Z ok (6.316s)
2022-09-27T16:04:16.5146490Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20226
2022-09-27T16:04:16.5152695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20227
2022-09-27T16:04:18.2037855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:18.2038409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:18.2039020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:18.2039525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:18.2102538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:18.2102987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:18.2105671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:18.2106165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:20.1958714Z dist init r=0, world=2
2022-09-27T16:04:20.1959064Z dist init r=1, world=2
2022-09-27T16:04:20.1964063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:20.1964828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:20.1965679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:20.1966401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:21.3397300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:21.3397856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:22.1100317Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:22.1101742Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:22.1102983Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:22.1104202Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:22.9293958Z ok (6.417s)
2022-09-27T16:04:22.9314395Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20312
2022-09-27T16:04:22.9320756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20313
2022-09-27T16:04:24.6095488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:24.6096055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:24.6096961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:24.6097443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:24.6741649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:24.6744025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:24.6744628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:24.6745107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:26.5604662Z dist init r=1, world=2
2022-09-27T16:04:26.5609221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:26.6283744Z dist init r=0, world=2
2022-09-27T16:04:26.6288390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:26.6289471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:26.6320171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:27.7634881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:27.7635431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:28.5170650Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:28.5172189Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:28.5173465Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:28.5174703Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:29.3459475Z ok (6.416s)
2022-09-27T16:04:29.3479841Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20398
2022-09-27T16:04:29.3486209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20399
2022-09-27T16:04:31.0436093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.0436630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.0437551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.0438009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.0633615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.0634096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.0637059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.0637524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:33.0293815Z dist init r=1, world=2
2022-09-27T16:04:33.0299140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:33.0525890Z dist init r=0, world=2
2022-09-27T16:04:33.0531718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:33.0533040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:33.0604606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:34.1852126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:34.1852648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:34.7931465Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:34.7932821Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:34.7934095Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:34.7935328Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:35.0283114Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:35.0284515Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:35.0285774Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:35.0287013Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:35.8628981Z ok (6.517s)
2022-09-27T16:04:35.8649430Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20484
2022-09-27T16:04:35.8656434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20485
2022-09-27T16:04:37.5686717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:37.5687268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:37.5687874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:37.5688373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:37.5851902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:37.5852382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:37.5856396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:37.5857023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:39.5735507Z dist init r=1, world=2
2022-09-27T16:04:39.5739943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:39.5756383Z dist init r=0, world=2
2022-09-27T16:04:39.5762455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:39.5764105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:39.5843320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:40.7178820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:40.7179520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:41.5118561Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:41.5119928Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:41.5121201Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:41.5122446Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:42.2795666Z ok (6.417s)
2022-09-27T16:04:42.2816638Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20570
2022-09-27T16:04:42.2822856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20571
2022-09-27T16:04:43.9894849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:43.9895541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:43.9896820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:43.9897482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:44.0221700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:44.0222532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:44.0225330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:44.0226184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:46.0062482Z dist init r=0, world=2
2022-09-27T16:04:46.0066806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:46.0096994Z dist init r=1, world=2
2022-09-27T16:04:46.0102976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:46.0104567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:46.0170300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:47.1586556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:47.1587122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:47.7702210Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:47.7703555Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:47.7704837Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:47.7706085Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:48.0161353Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:48.0162708Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:48.0164265Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:48.0165501Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:48.7958267Z ok (6.516s)
2022-09-27T16:04:48.7980336Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20656
2022-09-27T16:04:48.7987111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20657
2022-09-27T16:04:50.5073759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:50.5074892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:50.5075628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:50.5076122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:50.5336340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:50.5337088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:50.5340655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:50.5341315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:52.5192982Z dist init r=0, world=2
2022-09-27T16:04:52.5195683Z dist init r=1, world=2
2022-09-27T16:04:52.5196581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:52.5202713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:52.5204218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:52.5299739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:53.6538744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:53.6539464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:54.4343557Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:54.4344928Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:54.4346194Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:54.4347434Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:04:55.2124297Z ok (6.416s)
2022-09-27T16:04:55.2146244Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20742
2022-09-27T16:04:55.2152835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20743
2022-09-27T16:04:56.9346800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:56.9347329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:56.9347950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:56.9348431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.0428577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.0429100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.0429849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.0430329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:58.8596582Z dist init r=1, world=2
2022-09-27T16:04:58.8601076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:58.9827853Z dist init r=0, world=2
2022-09-27T16:04:58.9832895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:58.9833746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:58.9919152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:00.1174202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:00.1174829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:01.3282928Z ok (6.116s)
2022-09-27T16:05:01.3293218Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-09-27T16:05:01.3294220Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-09-27T16:05:01.3326775Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20828
2022-09-27T16:05:01.3333518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20829
2022-09-27T16:05:03.0118324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:03.0118903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:03.0119499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:03.0119988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:03.0765687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:03.0766191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:03.0767010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:03.0767507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.0460107Z dist init r=0, world=2
2022-09-27T16:05:05.0464632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:05.0502280Z dist init r=1, world=2
2022-09-27T16:05:05.0508385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:05.0509936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:05.0567890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:06.1994469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:06.1994991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:07.7476833Z ok (6.418s)
2022-09-27T16:05:07.7509194Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20914
2022-09-27T16:05:07.7515384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20915
2022-09-27T16:05:09.4747108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:09.4747606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:09.4748227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:09.4748712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:09.5133796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:09.5134284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:09.5137305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:09.5137795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:11.4740335Z dist init r=0, world=2
2022-09-27T16:05:11.4745382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:11.4814170Z dist init r=1, world=2
2022-09-27T16:05:11.4819596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:11.4820649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:11.4847987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:12.6080276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:12.6080816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:13.8651975Z ok (6.117s)
2022-09-27T16:05:13.8673042Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21000
2022-09-27T16:05:13.8679016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21001
2022-09-27T16:05:15.5704021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:15.5704732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:15.5705466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:15.5706068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:15.5729829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:15.5730551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:15.5733240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:15.5733964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:17.5415978Z dist init r=1, world=2
2022-09-27T16:05:17.5420157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:17.5728969Z dist init r=0, world=2
2022-09-27T16:05:17.5735666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:17.5736847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:17.5826574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:18.7140629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:18.7141203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:20.0813952Z ok (6.216s)
2022-09-27T16:05:20.0833110Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21086
2022-09-27T16:05:20.0839679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21087
2022-09-27T16:05:21.7390982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:21.7392001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:21.7393427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:21.7394418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:21.8007549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:21.8008448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:21.8009358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:21.8010444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:23.7024825Z dist init r=0, world=2
2022-09-27T16:05:23.7030459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:23.7631829Z dist init r=1, world=2
2022-09-27T16:05:23.7636629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:23.7638001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:23.7641444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:24.9331708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:24.9332508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:26.1991411Z ok (6.118s)
2022-09-27T16:05:26.2013556Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21172
2022-09-27T16:05:26.2020920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21173
2022-09-27T16:05:27.8921183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:27.8921720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:27.8922339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:27.8923154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:27.8923796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:27.8924265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:27.8927929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:27.8928428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:29.8938525Z dist init r=0, world=2
2022-09-27T16:05:29.8944080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:29.9021893Z dist init r=1, world=2
2022-09-27T16:05:29.9027573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:29.9028789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:29.9046873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:31.0383255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:31.0383803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:32.3156931Z ok (6.116s)
2022-09-27T16:05:32.3177228Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21258
2022-09-27T16:05:32.3183564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21259
2022-09-27T16:05:34.0093172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:34.0093771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:34.0094392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:34.0094885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:34.0457791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:34.0458279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:34.0460984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:34.0461481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:36.0254614Z dist init r=0, world=2
2022-09-27T16:05:36.0258728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:36.0495397Z dist init r=1, world=2
2022-09-27T16:05:36.0502350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:36.0503675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:36.0564653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:37.2278460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:37.2279002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:38.8327707Z ok (6.517s)
2022-09-27T16:05:38.8346622Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21344
2022-09-27T16:05:40.5610670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:40.5611502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:40.5612129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:40.5612612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:42.4791381Z dist init r=0, world=1
2022-09-27T16:05:42.4796246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:42.4797081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:05:42.5214912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:43.5440673Z ok (4.711s)
2022-09-27T16:05:43.5460367Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21387
2022-09-27T16:05:45.2338813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:45.2339491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:45.2340283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:45.2340840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:47.1178691Z dist init r=0, world=1
2022-09-27T16:05:47.1183625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:47.1184745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:05:47.1599300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:48.1551612Z ok (4.611s)
2022-09-27T16:05:48.1571750Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21430
2022-09-27T16:05:49.8172238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:49.8172934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:49.8173916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:49.8174422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:51.7087905Z dist init r=0, world=1
2022-09-27T16:05:51.7091874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:51.7093009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:05:51.7634810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:52.7662199Z ok (4.611s)
2022-09-27T16:05:52.7662393Z 
2022-09-27T16:05:52.7662890Z ----------------------------------------------------------------------
2022-09-27T16:05:52.7663173Z Ran 52 tests in 322.159s
2022-09-27T16:05:52.7663340Z 
2022-09-27T16:05:52.7663476Z OK (skipped=1)
2022-09-27T16:05:52.7666131Z 
2022-09-27T16:05:52.7666487Z Generating XML reports...
2022-09-27T16:05:52.7753612Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220927160030.xml
2022-09-27T16:05:52.7758490Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220927160030.xml
2022-09-27T16:05:53.2023187Z Running distributed/test_c10d_nccl ... [2022-09-27 16:05:53.201819]
2022-09-27T16:05:53.2023957Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:05:53.201925]
2022-09-27T16:05:55.1498602Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-09-27T16:05:55.1500768Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-09-27T16:05:55.1501112Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-09-27T16:05:55.1501428Z test_nccl_barrier (__main__.CommTest)
2022-09-27T16:05:55.1501764Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-09-27T16:05:55.1502129Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-09-27T16:05:55.1502806Z test_nccl_barrier_timeout (__main__.CommTest)
2022-09-27T16:05:55.1503180Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-09-27T16:05:55.1503555Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-09-27T16:05:55.1503910Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-09-27T16:05:55.1504300Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-09-27T16:05:55.1504664Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-09-27T16:05:55.1505257Z test_nncl_rank_membership (__main__.CommTest)
2022-09-27T16:05:55.1505821Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-09-27T16:05:55.1506497Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-09-27T16:05:55.1507165Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-09-27T16:05:55.1507672Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-09-27T16:05:55.1508122Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-09-27T16:05:55.1508465Z test_tensor_dtype_complex (__main__.CommTest)
2022-09-27T16:05:55.1508799Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-09-27T16:05:55.1509701Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-09-27T16:05:55.1510604Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1510967Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1511324Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1511673Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1512048Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-09-27T16:05:55.1512419Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1512898Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-09-27T16:05:55.1522134Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-09-27T16:05:55.1532035Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1532530Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1533005Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1533482Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1533928Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1534371Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1534811Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1535250Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1535702Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1536151Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1536638Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1537110Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1537600Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1538109Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1538620Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1539127Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1539621Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1540105Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1540588Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1541219Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1541736Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1542232Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1542724Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1543201Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1543693Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1544157Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1544634Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1545097Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1545529Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1559546Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1560061Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1560526Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1561000Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1561455Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1561971Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1562656Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1563215Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1563750Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1564298Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1564770Z test_fp16 (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1565185Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1565661Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1566112Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1566604Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1567085Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1567551Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1568073Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1568580Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1569116Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1569959Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1570660Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1571079Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1571543Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1572036Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1572524Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1573030Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1573632Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1574044Z test_no_grad (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1574440Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1574868Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1575300Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1575745Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1576202Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1576651Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-09-27T16:05:55.1577026Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:05:55.1578291Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-09-27T16:05:55.1579539Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1579963Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1580450Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1580852Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1581277Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1581694Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1582101Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1582484Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1582859Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-09-27T16:05:55.1583330Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-09-27T16:05:55.1583780Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-09-27T16:05:55.1585710Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2022-09-27T16:05:55.1587730Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1588193Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1588550Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1588915Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1589274Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1589611Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1589981Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1590347Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1590686Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1591047Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1591406Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1591786Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1592168Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1592556Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1592934Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1593278Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1593643Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1594005Z test_send_recv (__main__.ProcessGroupNCCLTest)
2022-09-27T16:05:55.1594420Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-09-27T16:05:55.1594845Z test_common_errors (__main__.RendezvousEnvTest)
2022-09-27T16:05:55.1595182Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:05:55.1595614Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-09-27T16:05:55.1596052Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-09-27T16:05:56.8027616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:56.8028429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:56.8031554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:56.8032078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:57.0957923Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:05:57.0971361Z 
2022-09-27T16:05:57.0971801Z Running tests...
2022-09-27T16:05:57.0972301Z ----------------------------------------------------------------------
2022-09-27T16:05:58.7545059Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:58.7745250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21541
2022-09-27T16:05:58.7751987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21542
2022-09-27T16:06:00.4467892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:00.4468748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:00.4469363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:00.4469832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:00.4600265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:00.4600746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:00.4603265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:00.4603737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:00.7313263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:00.7674712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:03.8869546Z ok (6.789s)
2022-09-27T16:06:03.8869819Z 
2022-09-27T16:06:03.8870216Z ----------------------------------------------------------------------
2022-09-27T16:06:03.8870577Z Ran 1 test in 6.790s
2022-09-27T16:06:03.8870753Z 
2022-09-27T16:06:03.8870856Z OK
2022-09-27T16:06:03.8870993Z 
2022-09-27T16:06:03.8871133Z Generating XML reports...
2022-09-27T16:06:03.8908264Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160557.xml
2022-09-27T16:06:05.9446522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:05.9447057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:05.9447672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:05.9448153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:06.2237103Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:06.2250811Z 
2022-09-27T16:06:06.2251148Z Running tests...
2022-09-27T16:06:06.2251593Z ----------------------------------------------------------------------
2022-09-27T16:06:07.8239884Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:07.8430674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21658
2022-09-27T16:06:07.8437591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21659
2022-09-27T16:06:09.5014190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:09.5015077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:09.5015774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:09.5016308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:09.5510571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:09.5511038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:09.5513395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:09.5513863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:09.7833294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:09.8264935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:12.9555051Z ok (6.730s)
2022-09-27T16:06:12.9555289Z 
2022-09-27T16:06:12.9555726Z ----------------------------------------------------------------------
2022-09-27T16:06:12.9556337Z Ran 1 test in 6.730s
2022-09-27T16:06:12.9556620Z 
2022-09-27T16:06:12.9556777Z OK
2022-09-27T16:06:12.9557027Z 
2022-09-27T16:06:12.9557256Z Generating XML reports...
2022-09-27T16:06:12.9596343Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160606.xml
2022-09-27T16:06:15.0567686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.0568196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.0568814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.0569304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.3473900Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:15.3488355Z 
2022-09-27T16:06:15.3488710Z Running tests...
2022-09-27T16:06:15.3489433Z ----------------------------------------------------------------------
2022-09-27T16:06:16.9805018Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:17.0004079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21775
2022-09-27T16:06:17.0010385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21776
2022-09-27T16:06:18.6716826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:18.6717316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:18.6719695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:18.6720210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:18.7133176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:18.7133644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:18.7136821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:18.7137308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:18.9620950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:18.9924614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:19.5073452Z skip: Need at least 4 CUDA devices (4.158s)
2022-09-27T16:06:19.5073735Z 
2022-09-27T16:06:19.5074158Z ----------------------------------------------------------------------
2022-09-27T16:06:19.5074811Z Ran 1 test in 4.158s
2022-09-27T16:06:19.5074978Z 
2022-09-27T16:06:19.5075094Z OK (skipped=1)
2022-09-27T16:06:19.5075233Z 
2022-09-27T16:06:19.5075365Z Generating XML reports...
2022-09-27T16:06:19.5112729Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160615.xml
2022-09-27T16:06:21.6765174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:21.6765718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:21.6766800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:21.6767283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:21.9743822Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:21.9758261Z 
2022-09-27T16:06:21.9758608Z Running tests...
2022-09-27T16:06:21.9759074Z ----------------------------------------------------------------------
2022-09-27T16:06:23.6202556Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:23.6402911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21878
2022-09-27T16:06:23.6409408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21879
2022-09-27T16:06:25.3006716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.3007256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.3007851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.3008342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.3162896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:25.3163384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:25.3165800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:25.3166646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:25.5986444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:25.5988937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:25.6030677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:25.6035077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:25.6036601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:25.6091932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:27.6504375Z ok (5.674s)
2022-09-27T16:06:27.6504611Z 
2022-09-27T16:06:27.6505066Z ----------------------------------------------------------------------
2022-09-27T16:06:27.6505426Z Ran 1 test in 5.674s
2022-09-27T16:06:27.6505599Z 
2022-09-27T16:06:27.6505698Z OK
2022-09-27T16:06:27.6505838Z 
2022-09-27T16:06:27.6505953Z Generating XML reports...
2022-09-27T16:06:27.6543371Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160621.xml
2022-09-27T16:06:29.7649085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:29.7650002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:29.7650919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:29.7651431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:30.0446195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:30.0460100Z 
2022-09-27T16:06:30.0460349Z Running tests...
2022-09-27T16:06:30.0460798Z ----------------------------------------------------------------------
2022-09-27T16:06:31.6432357Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:31.6625565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21994
2022-09-27T16:06:31.6631850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21995
2022-09-27T16:06:33.3157655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:33.3158178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:33.3158820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:33.3159302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:33.3570358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:33.3571097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:33.3573903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:33.3574387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:33.6036121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:33.6038732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:33.6244050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:33.6248087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:33.6249230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:33.6345417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:34.0691781Z ok (4.023s)
2022-09-27T16:06:34.0692000Z 
2022-09-27T16:06:34.0692426Z ----------------------------------------------------------------------
2022-09-27T16:06:34.0692781Z Ran 1 test in 4.023s
2022-09-27T16:06:34.0692897Z 
2022-09-27T16:06:34.0692996Z OK
2022-09-27T16:06:34.0693157Z 
2022-09-27T16:06:34.0693284Z Generating XML reports...
2022-09-27T16:06:34.0731220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160630.xml
2022-09-27T16:06:36.1562609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:36.1563195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:36.1563795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:36.1564259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:36.4374941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:36.4389034Z 
2022-09-27T16:06:36.4389386Z Running tests...
2022-09-27T16:06:36.4389834Z ----------------------------------------------------------------------
2022-09-27T16:06:38.0248958Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:38.0441016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22101
2022-09-27T16:06:38.0448522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22102
2022-09-27T16:06:39.7628587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:39.7629119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:39.7629728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:39.7630194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:39.8157398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:39.8158068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:39.8160457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:39.8160970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.0447648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:40.0888896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:40.5510998Z skip: Need at least 4 CUDA devices (4.112s)
2022-09-27T16:06:40.5511270Z 
2022-09-27T16:06:40.5511669Z ----------------------------------------------------------------------
2022-09-27T16:06:40.5512077Z Ran 1 test in 4.112s
2022-09-27T16:06:40.5512251Z 
2022-09-27T16:06:40.5512355Z OK (skipped=1)
2022-09-27T16:06:40.5512513Z 
2022-09-27T16:06:40.5512639Z Generating XML reports...
2022-09-27T16:06:40.5552061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160636.xml
2022-09-27T16:06:42.6866036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:42.6866574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:42.6867728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:42.6868531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:42.9795648Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:42.9810152Z 
2022-09-27T16:06:42.9810619Z Running tests...
2022-09-27T16:06:42.9811124Z ----------------------------------------------------------------------
2022-09-27T16:06:44.6093387Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:44.6296340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22204
2022-09-27T16:06:44.6303274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22205
2022-09-27T16:06:46.3561248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:46.3561780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:46.3562402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:46.3562831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:46.3586111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:46.3586583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:46.3589537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:46.3589995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:46.6397964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:46.6413438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:47.0363559Z skip: Need at least 4 CUDA devices (4.055s)
2022-09-27T16:06:47.0363856Z 
2022-09-27T16:06:47.0364284Z ----------------------------------------------------------------------
2022-09-27T16:06:47.0364625Z Ran 1 test in 4.055s
2022-09-27T16:06:47.0364766Z 
2022-09-27T16:06:47.0364876Z OK (skipped=1)
2022-09-27T16:06:47.0365040Z 
2022-09-27T16:06:47.0365170Z Generating XML reports...
2022-09-27T16:06:47.0403406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160642.xml
2022-09-27T16:06:49.1012699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:49.1013748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:49.1014989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:49.1015963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:49.3820058Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:49.3833166Z 
2022-09-27T16:06:49.3833422Z Running tests...
2022-09-27T16:06:49.3833857Z ----------------------------------------------------------------------
2022-09-27T16:06:50.9858676Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:51.0051987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22307
2022-09-27T16:06:51.0058550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22308
2022-09-27T16:06:52.6947766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:52.6948716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:52.6949883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:52.6950794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:52.6987457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:52.6988386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:52.6991845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:52.6992854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:52.9794955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:52.9859663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:53.4120383Z skip: Need at least 4 CUDA devices (4.028s)
2022-09-27T16:06:53.4120676Z 
2022-09-27T16:06:53.4121102Z ----------------------------------------------------------------------
2022-09-27T16:06:53.4121481Z Ran 1 test in 4.029s
2022-09-27T16:06:53.4121653Z 
2022-09-27T16:06:53.4121744Z OK (skipped=1)
2022-09-27T16:06:53.4121910Z 
2022-09-27T16:06:53.4122045Z Generating XML reports...
2022-09-27T16:06:53.4159403Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160649.xml
2022-09-27T16:06:55.5130587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:55.5131153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:55.5131763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:55.5132286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:55.8036179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:06:55.8050569Z 
2022-09-27T16:06:55.8050880Z Running tests...
2022-09-27T16:06:55.8051355Z ----------------------------------------------------------------------
2022-09-27T16:06:57.4447240Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:57.4646585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22410
2022-09-27T16:06:57.4653405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22411
2022-09-27T16:06:59.1647117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:59.1647664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:59.1648336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:59.1648813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:59.1835732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:59.1836277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:59.1838887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:59.1839362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:59.4630003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:59.4630583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:59.4842040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:59.4842623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:59.4843444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:59.4844439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:59.4845638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:06:59.4849416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:06:59.4850365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:59.4947548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:01.5750666Z ok (5.770s)
2022-09-27T16:07:01.5750971Z 
2022-09-27T16:07:01.5751450Z ----------------------------------------------------------------------
2022-09-27T16:07:01.5751800Z Ran 1 test in 5.770s
2022-09-27T16:07:01.5751938Z 
2022-09-27T16:07:01.5752016Z OK
2022-09-27T16:07:01.5752173Z 
2022-09-27T16:07:01.5752319Z Generating XML reports...
2022-09-27T16:07:01.5789744Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160655.xml
2022-09-27T16:07:03.6660632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:03.6661171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:03.6661758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:03.6662250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:03.9589350Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:03.9603149Z 
2022-09-27T16:07:03.9603711Z Running tests...
2022-09-27T16:07:03.9604183Z ----------------------------------------------------------------------
2022-09-27T16:07:05.5867297Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:05.6058626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22541
2022-09-27T16:07:05.6065414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22542
2022-09-27T16:07:07.2724719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:07.2725264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:07.2725892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:07.2726338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:07.2751076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:07.2751559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:07.2754280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:07.2754755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:07.5447464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:07.5449304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:07.5579010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:07.5582993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:07.5583823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:07.5586700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:07.5653931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:07.5654501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:07.5655552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:07.5689204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:09.6159573Z ok (5.655s)
2022-09-27T16:07:09.6159836Z 
2022-09-27T16:07:09.6160237Z ----------------------------------------------------------------------
2022-09-27T16:07:09.6160602Z Ran 1 test in 5.656s
2022-09-27T16:07:09.6160772Z 
2022-09-27T16:07:09.6160896Z OK
2022-09-27T16:07:09.6161048Z 
2022-09-27T16:07:09.6161159Z Generating XML reports...
2022-09-27T16:07:09.6199010Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160703.xml
2022-09-27T16:07:11.7309613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:11.7310119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:11.7310741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:11.7311228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:12.0254716Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:12.0269458Z 
2022-09-27T16:07:12.0269704Z Running tests...
2022-09-27T16:07:12.0270446Z ----------------------------------------------------------------------
2022-09-27T16:07:13.6747490Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:13.6948067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22663
2022-09-27T16:07:13.6954398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22664
2022-09-27T16:07:15.3662387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.3662912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.3663507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.3663994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.3884180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.3884684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.3887633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.3888138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.6678517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:15.6681421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:15.6755401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:15.6759505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:15.6760674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:15.6761566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:15.6783934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:15.6786865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:15.6787564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:15.6864887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:17.7053076Z ok (5.678s)
2022-09-27T16:07:17.7053333Z 
2022-09-27T16:07:17.7053745Z ----------------------------------------------------------------------
2022-09-27T16:07:17.7054101Z Ran 1 test in 5.678s
2022-09-27T16:07:17.7054275Z 
2022-09-27T16:07:17.7054384Z OK
2022-09-27T16:07:17.7054556Z 
2022-09-27T16:07:17.7054695Z Generating XML reports...
2022-09-27T16:07:17.7092968Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160712.xml
2022-09-27T16:07:19.7981511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:19.7982040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:19.7982631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:19.7983119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:20.0928958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:20.0944076Z 
2022-09-27T16:07:20.0944240Z Running tests...
2022-09-27T16:07:20.0944710Z ----------------------------------------------------------------------
2022-09-27T16:07:21.7435312Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:21.7635724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22785
2022-09-27T16:07:21.7641951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22786
2022-09-27T16:07:23.4920435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:23.4920965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:23.4921931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:23.4922466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:23.5218241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:23.5218676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:23.5221051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:23.5221553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:23.7786386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:23.7788975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:23.7995727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:23.8000736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:23.8001884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:23.8004462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:23.8095747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:23.8096313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:23.8097505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:23.8107066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:24.2703104Z ok (4.176s)
2022-09-27T16:07:24.2703282Z 
2022-09-27T16:07:24.2703723Z ----------------------------------------------------------------------
2022-09-27T16:07:24.2704051Z Ran 1 test in 4.176s
2022-09-27T16:07:24.2704229Z 
2022-09-27T16:07:24.2704274Z OK
2022-09-27T16:07:24.2704418Z 
2022-09-27T16:07:24.2704621Z Generating XML reports...
2022-09-27T16:07:24.2743958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160720.xml
2022-09-27T16:07:26.3765859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:26.3766416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:26.3767007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:26.3767493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:26.6565858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:26.6580503Z 
2022-09-27T16:07:26.6580641Z Running tests...
2022-09-27T16:07:26.6581114Z ----------------------------------------------------------------------
2022-09-27T16:07:28.2502147Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:28.2807948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22894
2022-09-27T16:07:28.2817036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22895
2022-09-27T16:07:29.9671598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:29.9672081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:29.9672675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:29.9673220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:29.9901598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:29.9902076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:29.9905161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:29.9905689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:30.2695256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:30.2697040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:30.2756865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:30.2761222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:30.2762487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:30.2765431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:30.2801056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:30.2803518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:30.2805058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:30.2870506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:33.4936202Z ok (6.835s)
2022-09-27T16:07:33.4936444Z 
2022-09-27T16:07:33.4936881Z ----------------------------------------------------------------------
2022-09-27T16:07:33.4937240Z Ran 1 test in 6.835s
2022-09-27T16:07:33.4937389Z 
2022-09-27T16:07:33.4937488Z OK
2022-09-27T16:07:33.4937627Z 
2022-09-27T16:07:33.4937769Z Generating XML reports...
2022-09-27T16:07:33.4976004Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160726.xml
2022-09-27T16:07:35.5733440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:35.5734246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:35.5734876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:35.5735625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:35.8572422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:35.8587235Z 
2022-09-27T16:07:35.8587693Z Running tests...
2022-09-27T16:07:35.8588245Z ----------------------------------------------------------------------
2022-09-27T16:07:37.4639080Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:37.4835958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23015
2022-09-27T16:07:37.4842367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23016
2022-09-27T16:07:39.1800776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:39.1801571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:39.1802140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:39.1802813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:39.2245251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:39.2245724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:39.2248305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:39.2248786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:39.4788687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:39.4799058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:39.4902517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:39.4913787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:39.4914965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:39.5003712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:39.5127502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:39.5127997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:39.5128684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:39.5130036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:41.5939824Z ok (5.735s)
2022-09-27T16:07:41.5940020Z 
2022-09-27T16:07:41.5940449Z ----------------------------------------------------------------------
2022-09-27T16:07:41.5940800Z Ran 1 test in 5.735s
2022-09-27T16:07:41.5940982Z 
2022-09-27T16:07:41.5941085Z OK
2022-09-27T16:07:41.5941212Z 
2022-09-27T16:07:41.5941332Z Generating XML reports...
2022-09-27T16:07:41.5980735Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160735.xml
2022-09-27T16:07:43.6892124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:43.6892719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:43.6893303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:43.6893784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:43.9676100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:43.9689528Z 
2022-09-27T16:07:43.9689741Z Running tests...
2022-09-27T16:07:43.9690485Z ----------------------------------------------------------------------
2022-09-27T16:07:45.5647593Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:45.5839674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23137
2022-09-27T16:07:45.5846735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23138
2022-09-27T16:07:47.2474604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:47.2475266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:47.2476789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:47.2477281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:47.2907579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:47.2908064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:47.2911328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:47.2911823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:47.5314571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:47.5687766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:47.9907771Z skip: Need at least 4 CUDA devices (4.021s)
2022-09-27T16:07:47.9908084Z 
2022-09-27T16:07:47.9908479Z ----------------------------------------------------------------------
2022-09-27T16:07:47.9908828Z Ran 1 test in 4.022s
2022-09-27T16:07:47.9909000Z 
2022-09-27T16:07:47.9909116Z OK (skipped=1)
2022-09-27T16:07:47.9909278Z 
2022-09-27T16:07:47.9909387Z Generating XML reports...
2022-09-27T16:07:47.9948283Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160743.xml
2022-09-27T16:07:50.0818441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:50.0818964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:50.0819593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:50.0820076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:50.3604465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:50.3617896Z 
2022-09-27T16:07:50.3618275Z Running tests...
2022-09-27T16:07:50.3618767Z ----------------------------------------------------------------------
2022-09-27T16:07:51.9618750Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:51.9812267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23240
2022-09-27T16:07:51.9819298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23241
2022-09-27T16:07:53.6244407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:53.6244975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:53.6246199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:53.6246742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:53.6524824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:53.6525224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:53.6528352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:53.6528866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:53.9139213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:53.9149091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:53.9336650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:53.9348359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:53.9349503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:53.9353589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:55.9911819Z ok (5.629s)
2022-09-27T16:07:55.9911985Z 
2022-09-27T16:07:55.9912407Z ----------------------------------------------------------------------
2022-09-27T16:07:55.9912744Z Ran 1 test in 5.629s
2022-09-27T16:07:55.9912890Z 
2022-09-27T16:07:55.9912991Z OK
2022-09-27T16:07:55.9913208Z 
2022-09-27T16:07:55.9913276Z Generating XML reports...
2022-09-27T16:07:55.9952011Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160750.xml
2022-09-27T16:07:58.0623188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:58.0623689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:58.0626702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:58.0627174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:58.3614875Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:07:58.3629189Z 
2022-09-27T16:07:58.3629689Z Running tests...
2022-09-27T16:07:58.3630241Z ----------------------------------------------------------------------
2022-09-27T16:07:59.9868545Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:00.0062242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23356
2022-09-27T16:08:00.0070093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23357
2022-09-27T16:08:01.6724942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:01.6725854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:01.6726514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:01.6726975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:01.7228103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:01.7229060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:01.7230202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:01.7230757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:01.9685101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:01.9696375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:02.0028713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:02.0040276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:02.0041411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:02.0044132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:02.0103933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:02.0106094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:02.0107246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:02.0146836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:04.1162825Z ok (5.753s)
2022-09-27T16:08:04.1163113Z 
2022-09-27T16:08:04.1163495Z ----------------------------------------------------------------------
2022-09-27T16:08:04.1163809Z Ran 1 test in 5.753s
2022-09-27T16:08:04.1163985Z 
2022-09-27T16:08:04.1164093Z OK
2022-09-27T16:08:04.1164252Z 
2022-09-27T16:08:04.1164378Z Generating XML reports...
2022-09-27T16:08:04.1203004Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160758.xml
2022-09-27T16:08:06.2208502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:06.2209016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:06.2209914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:06.2210496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:06.5002590Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:06.5017179Z 
2022-09-27T16:08:06.5017689Z Running tests...
2022-09-27T16:08:06.5018203Z ----------------------------------------------------------------------
2022-09-27T16:08:08.0937391Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:08.1131862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23476
2022-09-27T16:08:08.1139011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23477
2022-09-27T16:08:09.7895534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:09.7896042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:09.7896997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:09.7897485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:09.7909119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:09.7909570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:09.7912434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:09.7912919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:10.0713118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:10.0715194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:10.0847771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:10.0852219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:10.0853299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:10.0920318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:13.2252759Z ok (6.723s)
2022-09-27T16:08:13.2253016Z 
2022-09-27T16:08:13.2253450Z ----------------------------------------------------------------------
2022-09-27T16:08:13.2253800Z Ran 1 test in 6.723s
2022-09-27T16:08:13.2253969Z 
2022-09-27T16:08:13.2254054Z OK
2022-09-27T16:08:13.2254498Z 
2022-09-27T16:08:13.2254729Z Generating XML reports...
2022-09-27T16:08:13.2297560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160806.xml
2022-09-27T16:08:15.2950760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:15.2951299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:15.2951907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:15.2952373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:15.5770689Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:15.5784609Z 
2022-09-27T16:08:15.5784845Z Running tests...
2022-09-27T16:08:15.5785303Z ----------------------------------------------------------------------
2022-09-27T16:08:17.1607545Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:17.1799938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23593
2022-09-27T16:08:17.1806252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23594
2022-09-27T16:08:18.8397651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:18.8398235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:18.8399324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:18.8399840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:18.8573974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:18.8574458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:18.8576684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:18.8577518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:19.1411906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:19.1414353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:19.1473863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:19.1478080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:19.1479028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:19.1517599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:21.0899152Z ok (5.511s)
2022-09-27T16:08:21.0899434Z 
2022-09-27T16:08:21.0899923Z ----------------------------------------------------------------------
2022-09-27T16:08:21.0900268Z Ran 1 test in 5.511s
2022-09-27T16:08:21.0900417Z 
2022-09-27T16:08:21.0900517Z OK
2022-09-27T16:08:21.0900633Z 
2022-09-27T16:08:21.0900772Z Generating XML reports...
2022-09-27T16:08:21.0938025Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160815.xml
2022-09-27T16:08:23.2132403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:23.2132947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:23.2133530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:23.2134019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:23.4967986Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:23.4981850Z 
2022-09-27T16:08:23.4982099Z Running tests...
2022-09-27T16:08:23.4982534Z ----------------------------------------------------------------------
2022-09-27T16:08:25.1074829Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:25.1266477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23702
2022-09-27T16:08:25.1273187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23703
2022-09-27T16:08:26.8325857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:26.8326429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:26.8327038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:26.8327557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:26.8469617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:26.8470106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:26.8472690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:26.8473168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:27.1242817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:27.1364831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:28.5313523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:28.5754347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:28.5755258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:28.5819444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:29.7224658Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:29.7225437Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:29.7226341Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:29.7227071Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:30.3388617Z ok (6.840s)
2022-09-27T16:08:30.3388831Z 
2022-09-27T16:08:30.3389272Z ----------------------------------------------------------------------
2022-09-27T16:08:30.3389599Z Ran 1 test in 6.841s
2022-09-27T16:08:30.3389772Z 
2022-09-27T16:08:30.3389874Z OK
2022-09-27T16:08:30.3390015Z 
2022-09-27T16:08:30.3390150Z Generating XML reports...
2022-09-27T16:08:30.3427999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160823.xml
2022-09-27T16:08:32.4509161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:32.4509693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:32.4510602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:32.4511093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:32.7326387Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:32.7340946Z 
2022-09-27T16:08:32.7341552Z Running tests...
2022-09-27T16:08:32.7341997Z ----------------------------------------------------------------------
2022-09-27T16:08:34.3427943Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:34.3620074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23819
2022-09-27T16:08:34.3626801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23820
2022-09-27T16:08:36.0089306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:36.0090110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:36.0090752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:36.0091247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:36.0366258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:36.0366708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:36.0369718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:36.0370391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:36.2982776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:36.3404229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:37.7072153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:37.7378008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:37.7379081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:37.7478013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:38.9228682Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:38.9229559Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:38.9230868Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:38.9231671Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:39.4743441Z ok (6.740s)
2022-09-27T16:08:39.4743707Z 
2022-09-27T16:08:39.4744119Z ----------------------------------------------------------------------
2022-09-27T16:08:39.4744461Z Ran 1 test in 6.740s
2022-09-27T16:08:39.4744637Z 
2022-09-27T16:08:39.4744743Z OK
2022-09-27T16:08:39.4744902Z 
2022-09-27T16:08:39.4745068Z Generating XML reports...
2022-09-27T16:08:39.4782807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160832.xml
2022-09-27T16:08:41.5808430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:41.5809533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:41.5811630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:41.5812647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:41.8700265Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:41.8713177Z 
2022-09-27T16:08:41.8713724Z Running tests...
2022-09-27T16:08:41.8714248Z ----------------------------------------------------------------------
2022-09-27T16:08:43.4814625Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:43.5010092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23936
2022-09-27T16:08:43.5017879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23937
2022-09-27T16:08:45.1371833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:45.1372401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:45.1373003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:45.1373548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:45.2067451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:45.2067904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:45.2071496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:45.2071990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:45.4171867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:45.4773304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:46.7806563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:46.8388697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:46.8389573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:46.8414214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:48.0030248Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:48.0031133Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:48.0033793Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:48.0034540Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:48.6136287Z ok (6.742s)
2022-09-27T16:08:48.6136572Z 
2022-09-27T16:08:48.6136989Z ----------------------------------------------------------------------
2022-09-27T16:08:48.6137349Z Ran 1 test in 6.742s
2022-09-27T16:08:48.6137513Z 
2022-09-27T16:08:48.6137618Z OK
2022-09-27T16:08:48.6137798Z 
2022-09-27T16:08:48.6137946Z Generating XML reports...
2022-09-27T16:08:48.6176612Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160841.xml
2022-09-27T16:08:50.6760483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:50.6761313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:50.6761968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:50.6762446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:50.9655389Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:08:50.9670037Z 
2022-09-27T16:08:50.9670260Z Running tests...
2022-09-27T16:08:50.9670707Z ----------------------------------------------------------------------
2022-09-27T16:08:52.5942835Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:52.6143425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24053
2022-09-27T16:08:52.6149945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24054
2022-09-27T16:08:54.3139789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:54.3140317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:54.3141125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:54.3141594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:54.3158830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:54.3159293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:54.3161943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:54.3162414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:54.5986962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:54.6021895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:56.0134627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:56.0313435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:56.0314510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:56.0336607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:57.1893992Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1894854Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1895753Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1896475Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1897327Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1898042Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1899213Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1900023Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1900920Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1901693Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1902598Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1903366Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1904257Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1905035Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.1905939Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:08:57.1906714Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:08:57.7270305Z ok (6.760s)
2022-09-27T16:08:57.7270532Z 
2022-09-27T16:08:57.7270962Z ----------------------------------------------------------------------
2022-09-27T16:08:57.7271306Z Ran 1 test in 6.760s
2022-09-27T16:08:57.7271495Z 
2022-09-27T16:08:57.7271597Z OK
2022-09-27T16:08:57.7271714Z 
2022-09-27T16:08:57.7271850Z Generating XML reports...
2022-09-27T16:08:57.7311333Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160850.xml
2022-09-27T16:08:59.8108804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:59.8109307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:59.8109915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:59.8110403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:00.1051407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:00.1066111Z 
2022-09-27T16:09:00.1066255Z Running tests...
2022-09-27T16:09:00.1067000Z ----------------------------------------------------------------------
2022-09-27T16:09:01.7528626Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:01.7732586Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24170
2022-09-27T16:09:01.7738786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24171
2022-09-27T16:09:03.4990469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:03.4990996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:03.4991589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:03.4992033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:03.5021143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:03.5022002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:03.5025045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:03.5025563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:03.7710022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:03.8130551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:05.1311198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:05.1931052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:05.1931985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:05.2020438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:06.3686381Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:06.3687200Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:06.3688072Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:06.3688805Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:06.3689932Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:06.3690986Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:06.3691844Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:06.3692546Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:06.9861636Z ok (6.879s)
2022-09-27T16:09:06.9862093Z 
2022-09-27T16:09:06.9862851Z ----------------------------------------------------------------------
2022-09-27T16:09:06.9863259Z Ran 1 test in 6.879s
2022-09-27T16:09:06.9863432Z 
2022-09-27T16:09:06.9863533Z OK
2022-09-27T16:09:06.9863671Z 
2022-09-27T16:09:06.9863812Z Generating XML reports...
2022-09-27T16:09:06.9902925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160900.xml
2022-09-27T16:09:09.1047973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:09.1048517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:09.1049118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:09.1050029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:09.3866033Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:09.3879966Z 
2022-09-27T16:09:09.3880439Z Running tests...
2022-09-27T16:09:09.3880963Z ----------------------------------------------------------------------
2022-09-27T16:09:11.0047513Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:11.0239656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24287
2022-09-27T16:09:11.0246477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24288
2022-09-27T16:09:12.7097678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:12.7098208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:12.7098793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:12.7099277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:12.7519957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:12.7520447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:12.7523416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:12.7523926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:12.9939614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:13.0366238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:14.3719283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:14.3917692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:14.3918564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:14.3920981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:15.5565127Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:15.5566395Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:15.5567304Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:15.5568087Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:15.5569016Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:15.5570119Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:15.5571011Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:15.5571711Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:16.1366438Z ok (6.748s)
2022-09-27T16:09:16.1366877Z 
2022-09-27T16:09:16.1367348Z ----------------------------------------------------------------------
2022-09-27T16:09:16.1367708Z Ran 1 test in 6.749s
2022-09-27T16:09:16.1367875Z 
2022-09-27T16:09:16.1368024Z OK
2022-09-27T16:09:16.1368279Z 
2022-09-27T16:09:16.1368446Z Generating XML reports...
2022-09-27T16:09:16.1407652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160909.xml
2022-09-27T16:09:18.2381313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:18.2381877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:18.2382436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:18.2382923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:18.5269868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:18.5283431Z 
2022-09-27T16:09:18.5283731Z Running tests...
2022-09-27T16:09:18.5284204Z ----------------------------------------------------------------------
2022-09-27T16:09:20.1488192Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:20.1683072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24404
2022-09-27T16:09:20.1690073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24405
2022-09-27T16:09:21.7926518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:21.7927072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:21.7927633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:21.7928112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:21.8258985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:21.8259482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:21.8262205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:21.8262723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.0891099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:22.1257776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:23.5072217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:23.5349379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:23.5350187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:23.5375025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:24.7297492Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:24.7298354Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:24.7325448Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-09-27T16:09:24.7326146Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-09-27T16:09:25.3806805Z ok (6.852s)
2022-09-27T16:09:25.3807083Z 
2022-09-27T16:09:25.3807525Z ----------------------------------------------------------------------
2022-09-27T16:09:25.3807859Z Ran 1 test in 6.852s
2022-09-27T16:09:25.3807996Z 
2022-09-27T16:09:25.3808099Z OK
2022-09-27T16:09:25.3808238Z 
2022-09-27T16:09:25.3808664Z Generating XML reports...
2022-09-27T16:09:25.3854193Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160918.xml
2022-09-27T16:09:27.5255885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:27.5256459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:27.5257011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:27.5257498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:27.8091598Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:27.8107253Z 
2022-09-27T16:09:27.8107740Z Running tests...
2022-09-27T16:09:27.8108244Z ----------------------------------------------------------------------
2022-09-27T16:09:29.3967846Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:29.4159781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24523
2022-09-27T16:09:29.4165721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24524
2022-09-27T16:09:31.0754444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:31.0754959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:31.0755574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:31.0756062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:31.1004457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:31.1004922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:31.1007695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:31.1008557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:31.3662206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:31.4044742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:32.7704259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacuwjxg6
2022-09-27T16:09:32.7704924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacuwjxg6/_remote_module_non_scriptable.py
2022-09-27T16:09:32.7962571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o685_gi
2022-09-27T16:09:32.7965256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o685_gi/_remote_module_non_scriptable.py
2022-09-27T16:09:34.3861319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:34.3861892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:35.0296375Z ok (7.219s)
2022-09-27T16:09:35.0296632Z 
2022-09-27T16:09:35.0297065Z ----------------------------------------------------------------------
2022-09-27T16:09:35.0297499Z Ran 1 test in 7.219s
2022-09-27T16:09:35.0297816Z 
2022-09-27T16:09:35.0297955Z OK
2022-09-27T16:09:35.0298100Z 
2022-09-27T16:09:35.0298218Z Generating XML reports...
2022-09-27T16:09:35.0337678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160927.xml
2022-09-27T16:09:37.1384352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:37.1385100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:37.1385974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:37.1386489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:37.4136411Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:37.4150869Z 
2022-09-27T16:09:37.4151103Z Running tests...
2022-09-27T16:09:37.4151627Z ----------------------------------------------------------------------
2022-09-27T16:09:39.0414024Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:39.0614861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24644
2022-09-27T16:09:39.0621614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24645
2022-09-27T16:09:40.7108734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:40.7109377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:40.7110352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:40.7110882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:40.7418511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:40.7418998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:40.7421813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:40.7422314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:40.9980374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:41.0421691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:42.3830752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfohoo6t2
2022-09-27T16:09:42.3831730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfohoo6t2/_remote_module_non_scriptable.py
2022-09-27T16:09:42.4407843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkep87x9
2022-09-27T16:09:42.4409229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkep87x9/_remote_module_non_scriptable.py
2022-09-27T16:09:44.0830531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:44.0835750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:44.6743395Z ok (7.259s)
2022-09-27T16:09:44.6743650Z 
2022-09-27T16:09:44.6744077Z ----------------------------------------------------------------------
2022-09-27T16:09:44.6744435Z Ran 1 test in 7.259s
2022-09-27T16:09:44.6744607Z 
2022-09-27T16:09:44.6744714Z OK
2022-09-27T16:09:44.6744852Z 
2022-09-27T16:09:44.6744994Z Generating XML reports...
2022-09-27T16:09:44.6783367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160937.xml
2022-09-27T16:09:46.7314068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:46.7314614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:46.7315227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:46.7315719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:47.0055684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:47.0070061Z 
2022-09-27T16:09:47.0070454Z Running tests...
2022-09-27T16:09:47.0071270Z ----------------------------------------------------------------------
2022-09-27T16:09:48.6293859Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:48.6496588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24765
2022-09-27T16:09:48.6503677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24766
2022-09-27T16:09:50.2353123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.2353727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.2354798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.2355263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.3171434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.3172100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.3173214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.3173695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.4999217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:50.5917439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:51.8778386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgoyyson
2022-09-27T16:09:51.8779575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgoyyson/_remote_module_non_scriptable.py
2022-09-27T16:09:51.9732926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg696an6v
2022-09-27T16:09:51.9734943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg696an6v/_remote_module_non_scriptable.py
2022-09-27T16:09:53.6008426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:53.6009359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:54.2623731Z ok (7.255s)
2022-09-27T16:09:54.2624027Z 
2022-09-27T16:09:54.2624493Z ----------------------------------------------------------------------
2022-09-27T16:09:54.2624861Z Ran 1 test in 7.255s
2022-09-27T16:09:54.2625034Z 
2022-09-27T16:09:54.2625134Z OK
2022-09-27T16:09:54.2625255Z 
2022-09-27T16:09:54.2625397Z Generating XML reports...
2022-09-27T16:09:54.2664286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160947.xml
2022-09-27T16:09:56.3285548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:56.3286087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:56.3286701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:56.3287204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:56.6006808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:09:56.6022034Z 
2022-09-27T16:09:56.6022181Z Running tests...
2022-09-27T16:09:56.6023162Z ----------------------------------------------------------------------
2022-09-27T16:09:58.2353378Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:58.2562183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24886
2022-09-27T16:09:58.2567968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24887
2022-09-27T16:09:59.9150066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.9150546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.9152927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.9153395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:59.9222179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.9222642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.9225241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.9225696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:00.2004107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:00.2043370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:01.6075101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2vp0z3n
2022-09-27T16:10:01.6076074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2vp0z3n/_remote_module_non_scriptable.py
2022-09-27T16:10:01.6263285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxemx5hbz
2022-09-27T16:10:01.6266116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxemx5hbz/_remote_module_non_scriptable.py
2022-09-27T16:10:03.2212416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:10:03.2212926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:10:03.8696290Z ok (7.267s)
2022-09-27T16:10:03.8696510Z 
2022-09-27T16:10:03.8696946Z ----------------------------------------------------------------------
2022-09-27T16:10:03.8697313Z Ran 1 test in 7.267s
2022-09-27T16:10:03.8697498Z 
2022-09-27T16:10:03.8697587Z OK
2022-09-27T16:10:03.8697725Z 
2022-09-27T16:10:03.8698147Z Generating XML reports...
2022-09-27T16:10:03.8766223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160956.xml
2022-09-27T16:10:05.9944458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:05.9945020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:05.9945619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:05.9946084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:06.2756720Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:06.2770936Z 
2022-09-27T16:10:06.2771811Z Running tests...
2022-09-27T16:10:06.2772446Z ----------------------------------------------------------------------
2022-09-27T16:10:07.8698115Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:07.8890587Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25007
2022-09-27T16:10:07.8897222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25008
2022-09-27T16:10:09.5157299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.5157846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.5158458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.5158928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.5577595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.5578109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.5579858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.5580326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.7978346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:09.7979520Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:10:09.8423451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:09.8426029Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:10:11.1995400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8498dy8a
2022-09-27T16:10:11.1996303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8498dy8a/_remote_module_non_scriptable.py
2022-09-27T16:10:11.2523943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2bk72x4
2022-09-27T16:10:11.2526544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2bk72x4/_remote_module_non_scriptable.py
2022-09-27T16:10:13.0010299Z ok (6.723s)
2022-09-27T16:10:13.0010754Z 
2022-09-27T16:10:13.0011211Z ----------------------------------------------------------------------
2022-09-27T16:10:13.0011551Z Ran 1 test in 6.724s
2022-09-27T16:10:13.0011738Z 
2022-09-27T16:10:13.0011868Z OK
2022-09-27T16:10:13.0012284Z 
2022-09-27T16:10:13.0012425Z Generating XML reports...
2022-09-27T16:10:13.0050222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161006.xml
2022-09-27T16:10:15.0891178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:15.0891817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:15.0893996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:15.0894471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:15.3672972Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:15.3687152Z 
2022-09-27T16:10:15.3687470Z Running tests...
2022-09-27T16:10:15.3687916Z ----------------------------------------------------------------------
2022-09-27T16:10:16.9816761Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:17.0012540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25128
2022-09-27T16:10:17.0019921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25129
2022-09-27T16:10:18.6494894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.6495495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.6496355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.6496836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:18.6847385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.6847913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.6848508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.6848966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:18.9440356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:18.9441265Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:10:18.9890399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:18.9892727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:10:20.3498571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0m8yfq6
2022-09-27T16:10:20.3499750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0m8yfq6/_remote_module_non_scriptable.py
2022-09-27T16:10:20.4034653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47oggcmc
2022-09-27T16:10:20.4036217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47oggcmc/_remote_module_non_scriptable.py
2022-09-27T16:10:22.2139944Z ok (6.845s)
2022-09-27T16:10:22.2140144Z 
2022-09-27T16:10:22.2140599Z ----------------------------------------------------------------------
2022-09-27T16:10:22.2140932Z Ran 1 test in 6.845s
2022-09-27T16:10:22.2141173Z 
2022-09-27T16:10:22.2141501Z OK
2022-09-27T16:10:22.2141639Z 
2022-09-27T16:10:22.2141780Z Generating XML reports...
2022-09-27T16:10:22.2180449Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161015.xml
2022-09-27T16:10:24.3414246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:24.3414790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:24.3415420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:24.3415860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:24.6379849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:24.6395096Z 
2022-09-27T16:10:24.6395241Z Running tests...
2022-09-27T16:10:24.6395954Z ----------------------------------------------------------------------
2022-09-27T16:10:26.2882233Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:26.3083228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25249
2022-09-27T16:10:26.3089943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25250
2022-09-27T16:10:27.9644293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:27.9645387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:27.9646117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:27.9646897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.0124508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:28.0125442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:28.0128018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:28.0128819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.2412907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:28.3020361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:29.6363953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o0qmnp2
2022-09-27T16:10:29.6365180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o0qmnp2/_remote_module_non_scriptable.py
2022-09-27T16:10:29.7108538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_a5cvtm
2022-09-27T16:10:29.7110146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_a5cvtm/_remote_module_non_scriptable.py
2022-09-27T16:10:31.3206288Z ok (6.681s)
2022-09-27T16:10:31.3206735Z 
2022-09-27T16:10:31.3207245Z ----------------------------------------------------------------------
2022-09-27T16:10:31.3207606Z Ran 1 test in 6.681s
2022-09-27T16:10:31.3207778Z 
2022-09-27T16:10:31.3207853Z OK
2022-09-27T16:10:31.3207991Z 
2022-09-27T16:10:31.3208904Z Generating XML reports...
2022-09-27T16:10:31.3246095Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161024.xml
2022-09-27T16:10:33.4345579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:33.4346128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:33.4347055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:33.4347540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:33.7282676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:33.7298244Z 
2022-09-27T16:10:33.7298494Z Running tests...
2022-09-27T16:10:33.7298934Z ----------------------------------------------------------------------
2022-09-27T16:10:35.3509729Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:35.3704457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25370
2022-09-27T16:10:35.3712013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25371
2022-09-27T16:10:37.0216982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.0217537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.0218153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.0218644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.0411244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.0411717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.0415099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.0415568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.3139472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:37.3274263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:38.7442814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t0v7nsu
2022-09-27T16:10:38.7443482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t0v7nsu/_remote_module_non_scriptable.py
2022-09-27T16:10:38.7641260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5rm2n94
2022-09-27T16:10:38.7644096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5rm2n94/_remote_module_non_scriptable.py
2022-09-27T16:10:40.5834376Z ok (6.853s)
2022-09-27T16:10:40.5834852Z 
2022-09-27T16:10:40.5835552Z ----------------------------------------------------------------------
2022-09-27T16:10:40.5835888Z Ran 1 test in 6.853s
2022-09-27T16:10:40.5836062Z 
2022-09-27T16:10:40.5836162Z OK
2022-09-27T16:10:40.5836303Z 
2022-09-27T16:10:40.5836447Z Generating XML reports...
2022-09-27T16:10:40.5875486Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161033.xml
2022-09-27T16:10:42.6988639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:42.6989246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:42.6989897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:42.6990347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:42.9774441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:42.9788482Z 
2022-09-27T16:10:42.9789003Z Running tests...
2022-09-27T16:10:42.9789652Z ----------------------------------------------------------------------
2022-09-27T16:10:44.5928375Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:44.6121234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25491
2022-09-27T16:10:44.6128440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25492
2022-09-27T16:10:46.2594384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:46.2595066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:46.2596345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:46.2596912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:46.2833618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:46.2834470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:46.2837188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:46.2838014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:46.5547724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:46.5903904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:49.7246896Z ok (6.745s)
2022-09-27T16:10:49.7247321Z 
2022-09-27T16:10:49.7248110Z ----------------------------------------------------------------------
2022-09-27T16:10:49.7248584Z Ran 1 test in 6.746s
2022-09-27T16:10:49.7248738Z 
2022-09-27T16:10:49.7248835Z OK
2022-09-27T16:10:49.7248973Z 
2022-09-27T16:10:49.7249114Z Generating XML reports...
2022-09-27T16:10:49.7286387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161042.xml
2022-09-27T16:10:51.7975189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:51.7975772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:51.7976357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:51.7976863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:52.0838046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:10:52.0851076Z 
2022-09-27T16:10:52.0851305Z Running tests...
2022-09-27T16:10:52.0851770Z ----------------------------------------------------------------------
2022-09-27T16:10:52.0857540Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-09-27T16:10:53.6974568Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:53.7168208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25608
2022-09-27T16:10:53.7175165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25609
2022-09-27T16:10:55.4131337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:55.4131912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:55.4132529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:55.4133000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:55.4254913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:55.4255392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:55.4258173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:55.4258662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:55.7033978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:55.7090527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:57.1106859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu238hflc
2022-09-27T16:10:57.1107981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu238hflc/_remote_module_non_scriptable.py
2022-09-27T16:10:57.1446894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fkna5zy
2022-09-27T16:10:57.1447876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fkna5zy/_remote_module_non_scriptable.py
2022-09-27T16:10:58.3282389Z ok (6.243s)
2022-09-27T16:10:58.3282633Z 
2022-09-27T16:10:58.3283073Z ----------------------------------------------------------------------
2022-09-27T16:10:58.3283444Z Ran 1 test in 6.243s
2022-09-27T16:10:58.3283649Z 
2022-09-27T16:10:58.3283725Z OK
2022-09-27T16:10:58.3283870Z 
2022-09-27T16:10:58.3284012Z Generating XML reports...
2022-09-27T16:10:58.3322096Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161052.xml
2022-09-27T16:11:00.4262472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:00.4263074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:00.4263617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:00.4264079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:00.7191528Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:00.7206688Z 
2022-09-27T16:11:00.7207228Z Running tests...
2022-09-27T16:11:00.7208086Z ----------------------------------------------------------------------
2022-09-27T16:11:00.7213321Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T16:11:02.3781798Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:02.3983389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25728
2022-09-27T16:11:02.3989841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25729
2022-09-27T16:11:04.0628283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:04.0628807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:04.0629421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:04.0629913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:04.0815294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:04.0815769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:04.0818242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:04.0818749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:04.3643336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:04.3704702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:05.7794455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfsukhcu
2022-09-27T16:11:05.7795143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfsukhcu/_remote_module_non_scriptable.py
2022-09-27T16:11:05.8161207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cqsi8lx
2022-09-27T16:11:05.8163630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cqsi8lx/_remote_module_non_scriptable.py
2022-09-27T16:11:06.9097579Z ok (6.189s)
2022-09-27T16:11:06.9097854Z 
2022-09-27T16:11:06.9098268Z ----------------------------------------------------------------------
2022-09-27T16:11:06.9098601Z Ran 1 test in 6.189s
2022-09-27T16:11:06.9098770Z 
2022-09-27T16:11:06.9098870Z OK
2022-09-27T16:11:06.9099010Z 
2022-09-27T16:11:06.9099152Z Generating XML reports...
2022-09-27T16:11:06.9137041Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161100.xml
2022-09-27T16:11:08.9910583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:08.9911167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:08.9911802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:08.9912296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:09.2732803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:09.2747895Z 
2022-09-27T16:11:09.2748149Z Running tests...
2022-09-27T16:11:09.2748600Z ----------------------------------------------------------------------
2022-09-27T16:11:09.2755477Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:11:10.8674364Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:10.8867930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25848
2022-09-27T16:11:10.8874223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25849
2022-09-27T16:11:12.5631925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.5644495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.5645169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.5645699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.5735229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.5735727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.5737653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.5738128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.8582571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:12.8583146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:14.2993739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uk71nfl
2022-09-27T16:11:14.2994912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uk71nfl/_remote_module_non_scriptable.py
2022-09-27T16:11:14.3087916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsf5qv1j
2022-09-27T16:11:14.3091016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsf5qv1j/_remote_module_non_scriptable.py
2022-09-27T16:11:14.8433530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.8448742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.8768028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.8771597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.8923964Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:11:14.8925200Z   warnings.warn(
2022-09-27T16:11:14.8926253Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:11:14.8926998Z   warnings.warn(
2022-09-27T16:11:14.9030199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9035058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9242214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9244873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9537810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9541232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9793314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:14.9796734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:15.5982858Z ok (6.323s)
2022-09-27T16:11:15.5983085Z 
2022-09-27T16:11:15.5983838Z ----------------------------------------------------------------------
2022-09-27T16:11:15.5984232Z Ran 1 test in 6.323s
2022-09-27T16:11:15.5984380Z 
2022-09-27T16:11:15.5984483Z OK
2022-09-27T16:11:15.5984639Z 
2022-09-27T16:11:15.5984784Z Generating XML reports...
2022-09-27T16:11:15.6022662Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161109.xml
2022-09-27T16:11:17.6996367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:17.6996905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:17.6997546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:17.6998031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:17.9881962Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:17.9895883Z 
2022-09-27T16:11:17.9896186Z Running tests...
2022-09-27T16:11:17.9896624Z ----------------------------------------------------------------------
2022-09-27T16:11:17.9903713Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:11:19.6305811Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:19.6506918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25968
2022-09-27T16:11:19.6513384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25969
2022-09-27T16:11:21.2998371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:21.2998916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:21.2999543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:21.3000060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:21.3393365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:21.3394198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:21.3396524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:21.3397024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:21.5878918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:21.6341385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:22.9837226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd0b5c6i
2022-09-27T16:11:22.9837936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd0b5c6i/_remote_module_non_scriptable.py
2022-09-27T16:11:23.0239465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9ep6juz
2022-09-27T16:11:23.0240640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9ep6juz/_remote_module_non_scriptable.py
2022-09-27T16:11:23.5721451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.5722014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6049376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6050090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6206407Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:11:23.6207327Z   warnings.warn(
2022-09-27T16:11:23.6208429Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:11:23.6209167Z   warnings.warn(
2022-09-27T16:11:23.6315910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6316425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6530589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6531101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6837817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.6838519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.7097069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:23.7097826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:24.3624185Z ok (6.372s)
2022-09-27T16:11:24.3624426Z 
2022-09-27T16:11:24.3624856Z ----------------------------------------------------------------------
2022-09-27T16:11:24.3625216Z Ran 1 test in 6.373s
2022-09-27T16:11:24.3625396Z 
2022-09-27T16:11:24.3625472Z OK
2022-09-27T16:11:24.3625615Z 
2022-09-27T16:11:24.3625751Z Generating XML reports...
2022-09-27T16:11:24.3664026Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161117.xml
2022-09-27T16:11:26.4375087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:26.4375996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:26.4376673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:26.4377202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:26.7202958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:26.7216735Z 
2022-09-27T16:11:26.7217146Z Running tests...
2022-09-27T16:11:26.7217650Z ----------------------------------------------------------------------
2022-09-27T16:11:26.7223153Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:11:28.3382299Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:28.3576537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26088
2022-09-27T16:11:28.3583494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26089
2022-09-27T16:11:30.0187198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:30.0187961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:30.0188559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:30.0189027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:30.0487242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:30.0488209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:30.0490549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:30.0491032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:30.3161365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:30.3548996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:31.7225215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcx4x0trv
2022-09-27T16:11:31.7226319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcx4x0trv/_remote_module_non_scriptable.py
2022-09-27T16:11:31.7728464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82kag4lu
2022-09-27T16:11:31.7730147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82kag4lu/_remote_module_non_scriptable.py
2022-09-27T16:11:32.3201359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:32.3203315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:32.3509602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:32.3513242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:32.9690859Z ok (6.247s)
2022-09-27T16:11:32.9691104Z 
2022-09-27T16:11:32.9691522Z ----------------------------------------------------------------------
2022-09-27T16:11:32.9691888Z Ran 1 test in 6.247s
2022-09-27T16:11:32.9692062Z 
2022-09-27T16:11:32.9692165Z OK
2022-09-27T16:11:32.9692306Z 
2022-09-27T16:11:32.9692450Z Generating XML reports...
2022-09-27T16:11:32.9734561Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161126.xml
2022-09-27T16:11:35.0818653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:35.0819411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:35.0820536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:35.0821266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.3782121Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:35.3796266Z 
2022-09-27T16:11:35.3796695Z Running tests...
2022-09-27T16:11:35.3797210Z ----------------------------------------------------------------------
2022-09-27T16:11:35.3802621Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:11:37.0230264Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:37.0421637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26208
2022-09-27T16:11:37.0428814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26209
2022-09-27T16:11:38.7236105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.7237155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.7238372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.7239323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.7635649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.7636174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.7639178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.7639678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:39.0139525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:39.0591451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:40.4147658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yh1ty8j
2022-09-27T16:11:40.4148312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yh1ty8j/_remote_module_non_scriptable.py
2022-09-27T16:11:40.4609190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jrnvofb
2022-09-27T16:11:40.4610876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jrnvofb/_remote_module_non_scriptable.py
2022-09-27T16:11:41.0270078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:41.0273803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:41.0610690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:41.0617619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:41.6536219Z ok (6.274s)
2022-09-27T16:11:41.6536522Z 
2022-09-27T16:11:41.6536878Z ----------------------------------------------------------------------
2022-09-27T16:11:41.6537176Z Ran 1 test in 6.274s
2022-09-27T16:11:41.6537358Z 
2022-09-27T16:11:41.6537471Z OK
2022-09-27T16:11:41.6537590Z 
2022-09-27T16:11:41.6537706Z Generating XML reports...
2022-09-27T16:11:41.6574989Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161135.xml
2022-09-27T16:11:43.8130522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.8131012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.8131698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.8132444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:44.0908663Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:44.0922157Z 
2022-09-27T16:11:44.0922376Z Running tests...
2022-09-27T16:11:44.0923338Z ----------------------------------------------------------------------
2022-09-27T16:11:44.0931288Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:11:45.6866176Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:45.7059165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26328
2022-09-27T16:11:45.7065496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26329
2022-09-27T16:11:47.4152201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.4152793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.4153405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.4153809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.4288565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.4289050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.4291752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.4292466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.7101739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:47.7159101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:49.1805288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiz6vjoa
2022-09-27T16:11:49.1805946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiz6vjoa/_remote_module_non_scriptable.py
2022-09-27T16:11:49.1822917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput4986s_
2022-09-27T16:11:49.1826128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput4986s_/_remote_module_non_scriptable.py
2022-09-27T16:11:49.7464768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:49.7465290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:49.7740890Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:11:49.7742587Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:11:49.8106659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:49.8108280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:50.5176153Z ok (6.425s)
2022-09-27T16:11:50.5176403Z 
2022-09-27T16:11:50.5176825Z ----------------------------------------------------------------------
2022-09-27T16:11:50.5177191Z Ran 1 test in 6.425s
2022-09-27T16:11:50.5177367Z 
2022-09-27T16:11:50.5177473Z OK
2022-09-27T16:11:50.5177619Z 
2022-09-27T16:11:50.5177762Z Generating XML reports...
2022-09-27T16:11:50.5216524Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161144.xml
2022-09-27T16:11:52.5875935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:52.5876482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:52.5877113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:52.5877535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:52.8793432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:11:52.8808261Z 
2022-09-27T16:11:52.8808731Z Running tests...
2022-09-27T16:11:52.8809206Z ----------------------------------------------------------------------
2022-09-27T16:11:52.8818881Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:11:54.5155297Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:54.5355260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26448
2022-09-27T16:11:54.5362292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26449
2022-09-27T16:11:56.2128019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:56.2129085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:56.2130677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:56.2131660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:56.2171306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:56.2171821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:56.2174107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:56.2174635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:56.4861685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:56.5311286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:57.8682214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzmy5rbk
2022-09-27T16:11:57.8683295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzmy5rbk/_remote_module_non_scriptable.py
2022-09-27T16:11:57.9429903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppscvry7i
2022-09-27T16:11:57.9431079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppscvry7i/_remote_module_non_scriptable.py
2022-09-27T16:11:59.1470292Z ok (6.266s)
2022-09-27T16:11:59.1470533Z 
2022-09-27T16:11:59.1470969Z ----------------------------------------------------------------------
2022-09-27T16:11:59.1471324Z Ran 1 test in 6.266s
2022-09-27T16:11:59.1471449Z 
2022-09-27T16:11:59.1471570Z OK
2022-09-27T16:11:59.1471714Z 
2022-09-27T16:11:59.1471858Z Generating XML reports...
2022-09-27T16:11:59.1511483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161152.xml
2022-09-27T16:12:01.2653284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:01.2653845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:01.2654468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:01.2654928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:01.5521892Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:01.5536563Z 
2022-09-27T16:12:01.5536886Z Running tests...
2022-09-27T16:12:01.5537350Z ----------------------------------------------------------------------
2022-09-27T16:12:01.5542991Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-09-27T16:12:03.1555034Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:03.1796880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26568
2022-09-27T16:12:03.1801884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26569
2022-09-27T16:12:04.8207468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.8208053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.8208653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.8209090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.8482433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.8482970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.8485126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.8485604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:05.1063394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:05.1503518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:06.5101942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxymbmi3r
2022-09-27T16:12:06.5102605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxymbmi3r/_remote_module_non_scriptable.py
2022-09-27T16:12:06.5632589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p0q2re6
2022-09-27T16:12:06.5633983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p0q2re6/_remote_module_non_scriptable.py
2022-09-27T16:12:07.1095060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:07.1097951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:07.1412161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:07.1418228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:07.6906727Z ok (6.137s)
2022-09-27T16:12:07.6906944Z 
2022-09-27T16:12:07.6907374Z ----------------------------------------------------------------------
2022-09-27T16:12:07.6907775Z Ran 1 test in 6.137s
2022-09-27T16:12:07.6907939Z 
2022-09-27T16:12:07.6908025Z OK
2022-09-27T16:12:07.6908164Z 
2022-09-27T16:12:07.6908282Z Generating XML reports...
2022-09-27T16:12:07.6946515Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161201.xml
2022-09-27T16:12:09.7663100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.7663646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.7664236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.7664727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:10.0568295Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:10.0582705Z 
2022-09-27T16:12:10.0583258Z Running tests...
2022-09-27T16:12:10.0583725Z ----------------------------------------------------------------------
2022-09-27T16:12:10.0591755Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:12:11.6691025Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:11.6883898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26688
2022-09-27T16:12:11.6890645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26689
2022-09-27T16:12:13.3220237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.3220790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.3221380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.3221896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.3487117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.3487924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.3490339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.3490801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.6133870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:13.6169748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:15.0291145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx5vu5lg
2022-09-27T16:12:15.0292200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx5vu5lg/_remote_module_non_scriptable.py
2022-09-27T16:12:15.0368295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15631mxq
2022-09-27T16:12:15.0370767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15631mxq/_remote_module_non_scriptable.py
2022-09-27T16:12:15.5697581Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:12:15.5699284Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:12:15.5981513Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:12:15.5982296Z   warnings.warn(
2022-09-27T16:12:15.5983363Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:12:15.5984112Z   warnings.warn(
2022-09-27T16:12:15.6090679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:15.6091177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:15.6593750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:15.6594260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:16.2996040Z ok (6.241s)
2022-09-27T16:12:16.2996326Z 
2022-09-27T16:12:16.2996711Z ----------------------------------------------------------------------
2022-09-27T16:12:16.2997088Z Ran 1 test in 6.241s
2022-09-27T16:12:16.2997275Z 
2022-09-27T16:12:16.2997322Z OK
2022-09-27T16:12:16.2997458Z 
2022-09-27T16:12:16.2997604Z Generating XML reports...
2022-09-27T16:12:16.3037061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161210.xml
2022-09-27T16:12:18.3994703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.3995278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.3995839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.3996351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:18.6898574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:18.6912901Z 
2022-09-27T16:12:18.6913359Z Running tests...
2022-09-27T16:12:18.6913881Z ----------------------------------------------------------------------
2022-09-27T16:12:18.6922759Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:12:20.3369363Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:20.3570105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26808
2022-09-27T16:12:20.3577190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26809
2022-09-27T16:12:22.0094346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:22.0094904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:22.0095973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:22.0096534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:22.0254576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:22.0254992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:22.0257682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:22.0258163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:22.3078573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:22.3129926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:23.7213222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5bh87k7
2022-09-27T16:12:23.7213844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5bh87k7/_remote_module_non_scriptable.py
2022-09-27T16:12:23.7518761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_in4s72
2022-09-27T16:12:23.7521758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_in4s72/_remote_module_non_scriptable.py
2022-09-27T16:12:24.2974813Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:12:24.2975678Z   warnings.warn(
2022-09-27T16:12:24.2976752Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:12:24.2977488Z   warnings.warn(
2022-09-27T16:12:24.3115330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:24.3120287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:24.3519319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:24.3523870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:24.9680731Z ok (6.276s)
2022-09-27T16:12:24.9681009Z 
2022-09-27T16:12:24.9681409Z ----------------------------------------------------------------------
2022-09-27T16:12:24.9681769Z Ran 1 test in 6.277s
2022-09-27T16:12:24.9681947Z 
2022-09-27T16:12:24.9682053Z OK
2022-09-27T16:12:24.9682198Z 
2022-09-27T16:12:24.9682343Z Generating XML reports...
2022-09-27T16:12:24.9721562Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161218.xml
2022-09-27T16:12:27.0572008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:27.0572579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:27.0573177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:27.0573688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:27.3355383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:27.3368700Z 
2022-09-27T16:12:27.3368948Z Running tests...
2022-09-27T16:12:27.3369406Z ----------------------------------------------------------------------
2022-09-27T16:12:27.3379299Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-09-27T16:12:28.9445120Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:28.9637569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26928
2022-09-27T16:12:28.9643801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26929
2022-09-27T16:12:30.6169357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.6170491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.6171086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.6171574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.6458501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.6458996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.6461939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.6462435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.9060085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:30.9506207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:32.2960784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_c7zjzih
2022-09-27T16:12:32.2961440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_c7zjzih/_remote_module_non_scriptable.py
2022-09-27T16:12:32.3461306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywngnknq
2022-09-27T16:12:32.3463112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywngnknq/_remote_module_non_scriptable.py
2022-09-27T16:12:32.8937286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:32.8937837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:32.9308165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:32.9308734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:33.5751525Z ok (6.238s)
2022-09-27T16:12:33.5751788Z 
2022-09-27T16:12:33.5752236Z ----------------------------------------------------------------------
2022-09-27T16:12:33.5752609Z Ran 1 test in 6.238s
2022-09-27T16:12:33.5752783Z 
2022-09-27T16:12:33.5752857Z OK
2022-09-27T16:12:33.5753010Z 
2022-09-27T16:12:33.5753158Z Generating XML reports...
2022-09-27T16:12:33.5791665Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161227.xml
2022-09-27T16:12:35.6464003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.6464558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.6465196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.6465669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.9258987Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:35.9271870Z 
2022-09-27T16:12:35.9271981Z Running tests...
2022-09-27T16:12:35.9272847Z ----------------------------------------------------------------------
2022-09-27T16:12:35.9282072Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-09-27T16:12:37.5315796Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:37.5508404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27048
2022-09-27T16:12:37.5515763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27049
2022-09-27T16:12:39.2022335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.2022893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.2023828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.2024330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.2298061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.2298506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.2301482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.2301944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.4958787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:39.5332411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:40.9024345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifx2sy6y
2022-09-27T16:12:40.9025310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifx2sy6y/_remote_module_non_scriptable.py
2022-09-27T16:12:40.9369223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshh_lt1e
2022-09-27T16:12:40.9371796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshh_lt1e/_remote_module_non_scriptable.py
2022-09-27T16:12:41.4736655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.4753997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5059878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5063124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5277015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5292029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5625662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:41.5634946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:42.1622341Z ok (6.235s)
2022-09-27T16:12:42.1622583Z 
2022-09-27T16:12:42.1623029Z ----------------------------------------------------------------------
2022-09-27T16:12:42.1623361Z Ran 1 test in 6.235s
2022-09-27T16:12:42.1623534Z 
2022-09-27T16:12:42.1623632Z OK
2022-09-27T16:12:42.1623773Z 
2022-09-27T16:12:42.1623915Z Generating XML reports...
2022-09-27T16:12:42.1662966Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161235.xml
2022-09-27T16:12:44.2745594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:44.2746092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:44.2746779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:44.2747235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:44.5670254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:44.5685155Z 
2022-09-27T16:12:44.5685367Z Running tests...
2022-09-27T16:12:44.5686891Z ----------------------------------------------------------------------
2022-09-27T16:12:46.2005054Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:46.2197510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27168
2022-09-27T16:12:46.2205594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27169
2022-09-27T16:12:47.8177925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.8178463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.8179059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.8179556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.8995888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:47.8996427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:47.8997287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:47.8997785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:48.0842916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:48.1784641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:49.4570694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtgql1gl
2022-09-27T16:12:49.4571345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtgql1gl/_remote_module_non_scriptable.py
2022-09-27T16:12:49.6044052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu53_gi35
2022-09-27T16:12:49.6045108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu53_gi35/_remote_module_non_scriptable.py
2022-09-27T16:12:51.2339526Z ok (6.665s)
2022-09-27T16:12:51.2339824Z 
2022-09-27T16:12:51.2340241Z ----------------------------------------------------------------------
2022-09-27T16:12:51.2340917Z Ran 1 test in 6.665s
2022-09-27T16:12:51.2341114Z 
2022-09-27T16:12:51.2341193Z OK
2022-09-27T16:12:51.2341336Z 
2022-09-27T16:12:51.2341481Z Generating XML reports...
2022-09-27T16:12:51.2378966Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161244.xml
2022-09-27T16:12:53.3079173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:53.3079713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:53.3080340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:53.3080808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:53.5882381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:12:53.5896536Z 
2022-09-27T16:12:53.5896899Z Running tests...
2022-09-27T16:12:53.5897345Z ----------------------------------------------------------------------
2022-09-27T16:12:55.1768517Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:55.1961489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27289
2022-09-27T16:12:55.1967848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27290
2022-09-27T16:12:56.8986564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:56.8987095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:56.8987718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:56.8988202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:56.9105489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:56.9105950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:56.9108702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:56.9109200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:57.1865693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:57.1923653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:58.5868527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuljuw838
2022-09-27T16:12:58.5869167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuljuw838/_remote_module_non_scriptable.py
2022-09-27T16:12:58.6377264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw73hdeje
2022-09-27T16:12:58.6378502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw73hdeje/_remote_module_non_scriptable.py
2022-09-27T16:13:00.4089224Z ok (6.819s)
2022-09-27T16:13:00.4089464Z 
2022-09-27T16:13:00.4090122Z ----------------------------------------------------------------------
2022-09-27T16:13:00.4090496Z Ran 1 test in 6.819s
2022-09-27T16:13:00.4090646Z 
2022-09-27T16:13:00.4090743Z OK
2022-09-27T16:13:00.4090886Z 
2022-09-27T16:13:00.4090995Z Generating XML reports...
2022-09-27T16:13:00.4128835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161253.xml
2022-09-27T16:13:02.5040059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:02.5040694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:02.5041641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:02.5042128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:02.7858941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:02.7873325Z 
2022-09-27T16:13:02.7874006Z Running tests...
2022-09-27T16:13:02.7874518Z ----------------------------------------------------------------------
2022-09-27T16:13:04.3964548Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:04.4156997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27410
2022-09-27T16:13:04.4163104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27411
2022-09-27T16:13:06.1268151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:06.1268727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:06.1269600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:06.1270115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:06.1418296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:06.1418765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:06.1421903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:06.1422406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:06.4209799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:06.4210427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:07.8308639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisasxlir
2022-09-27T16:13:07.8309813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisasxlir/_remote_module_non_scriptable.py
2022-09-27T16:13:07.8472648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzecnncco
2022-09-27T16:13:07.8475517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzecnncco/_remote_module_non_scriptable.py
2022-09-27T16:13:09.6281730Z ok (6.841s)
2022-09-27T16:13:09.6282000Z 
2022-09-27T16:13:09.6282441Z ----------------------------------------------------------------------
2022-09-27T16:13:09.6282789Z Ran 1 test in 6.841s
2022-09-27T16:13:09.6282939Z 
2022-09-27T16:13:09.6283040Z OK
2022-09-27T16:13:09.6283182Z 
2022-09-27T16:13:09.6283323Z Generating XML reports...
2022-09-27T16:13:09.6322134Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161302.xml
2022-09-27T16:13:11.7133446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:11.7133969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:11.7134524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:11.7135008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:11.9932275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:11.9946408Z 
2022-09-27T16:13:11.9946538Z Running tests...
2022-09-27T16:13:11.9947512Z ----------------------------------------------------------------------
2022-09-27T16:13:11.9957189Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:13:13.5917175Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:13.6109635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27531
2022-09-27T16:13:13.6116496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27532
2022-09-27T16:13:15.2973636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:15.2974157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:15.2974778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:15.2975239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:15.3160799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:15.3161226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:15.3163732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:15.3164204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:15.5879729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:15.6014949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:16.9944570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0esjkhwm
2022-09-27T16:13:16.9945219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0esjkhwm/_remote_module_non_scriptable.py
2022-09-27T16:13:17.0204697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9boanbcs
2022-09-27T16:13:17.0207641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9boanbcs/_remote_module_non_scriptable.py
2022-09-27T16:13:18.8231949Z ok (6.828s)
2022-09-27T16:13:18.8232201Z 
2022-09-27T16:13:18.8232684Z ----------------------------------------------------------------------
2022-09-27T16:13:18.8233017Z Ran 1 test in 6.828s
2022-09-27T16:13:18.8233501Z 
2022-09-27T16:13:18.8233604Z OK
2022-09-27T16:13:18.8233744Z 
2022-09-27T16:13:18.8233885Z Generating XML reports...
2022-09-27T16:13:18.8272608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161311.xml
2022-09-27T16:13:20.8847361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:20.8847940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:20.8848528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:20.8848981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:21.1769097Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:21.1784189Z 
2022-09-27T16:13:21.1784330Z Running tests...
2022-09-27T16:13:21.1785124Z ----------------------------------------------------------------------
2022-09-27T16:13:21.1790836Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-09-27T16:13:22.8032534Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:22.8224685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27652
2022-09-27T16:13:22.8231891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27653
2022-09-27T16:13:24.5106967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:24.5107510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:24.5108431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:24.5108935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:24.5311021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:24.5311493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:24.5314470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:24.5314962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:24.8082803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:24.8216540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:26.2170889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpwd5cct
2022-09-27T16:13:26.2171546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpwd5cct/_remote_module_non_scriptable.py
2022-09-27T16:13:26.2519761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qmpubju
2022-09-27T16:13:26.2520933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qmpubju/_remote_module_non_scriptable.py
2022-09-27T16:13:28.0347741Z ok (6.856s)
2022-09-27T16:13:28.0347999Z 
2022-09-27T16:13:28.0348443Z ----------------------------------------------------------------------
2022-09-27T16:13:28.0349022Z Ran 1 test in 6.856s
2022-09-27T16:13:28.0349324Z 
2022-09-27T16:13:28.0349446Z OK
2022-09-27T16:13:28.0349614Z 
2022-09-27T16:13:28.0349852Z Generating XML reports...
2022-09-27T16:13:28.0389113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161321.xml
2022-09-27T16:13:30.1417251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:30.1417766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:30.1418372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:30.1419125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:30.4174198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:30.4187993Z 
2022-09-27T16:13:30.4188312Z Running tests...
2022-09-27T16:13:30.4188744Z ----------------------------------------------------------------------
2022-09-27T16:13:32.0249134Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:32.0440297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27773
2022-09-27T16:13:32.0447055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27774
2022-09-27T16:13:33.6934158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:33.6934850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:33.6936280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:33.6937179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:33.7286320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:33.7286832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:33.7289500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:33.7290237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:33.9861164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:34.0115710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:34.4507569Z skip: Need at least 4 CUDA devices (4.032s)
2022-09-27T16:13:34.4507838Z 
2022-09-27T16:13:34.4508262Z ----------------------------------------------------------------------
2022-09-27T16:13:34.4508618Z Ran 1 test in 4.032s
2022-09-27T16:13:34.4508765Z 
2022-09-27T16:13:34.4508885Z OK (skipped=1)
2022-09-27T16:13:34.4509045Z 
2022-09-27T16:13:34.4509182Z Generating XML reports...
2022-09-27T16:13:34.4547698Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161330.xml
2022-09-27T16:13:36.5866418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:36.5867018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:36.5867591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:36.5868085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:36.8698373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:36.8712855Z 
2022-09-27T16:13:36.8713162Z Running tests...
2022-09-27T16:13:36.8713585Z ----------------------------------------------------------------------
2022-09-27T16:13:38.4706268Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:38.4899565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27876
2022-09-27T16:13:38.4906206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27877
2022-09-27T16:13:40.1369530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:40.1370362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:40.1371393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:40.1372265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:40.1644667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:40.1645155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:40.1647890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:40.1648383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:40.4283987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:40.4407713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:41.9022592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fks_lly
2022-09-27T16:13:41.9023321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fks_lly/_remote_module_non_scriptable.py
2022-09-27T16:13:41.9129561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrzxi7v3
2022-09-27T16:13:41.9132952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrzxi7v3/_remote_module_non_scriptable.py
2022-09-27T16:13:43.0747313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.0762452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.1318628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.1335014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.1876187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.1888324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.2408889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.2422708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:13:43.9029714Z ok (7.031s)
2022-09-27T16:13:43.9029952Z 
2022-09-27T16:13:43.9030398Z ----------------------------------------------------------------------
2022-09-27T16:13:43.9030724Z Ran 1 test in 7.032s
2022-09-27T16:13:43.9030899Z 
2022-09-27T16:13:43.9031003Z OK
2022-09-27T16:13:43.9031144Z 
2022-09-27T16:13:43.9031281Z Generating XML reports...
2022-09-27T16:13:43.9070249Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161336.xml
2022-09-27T16:13:45.9810950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:45.9811455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:45.9812139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:45.9812560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:46.2635341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:46.2649179Z 
2022-09-27T16:13:46.2649409Z Running tests...
2022-09-27T16:13:46.2650301Z ----------------------------------------------------------------------
2022-09-27T16:13:47.8802897Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:47.8996139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27997
2022-09-27T16:13:47.9002926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27998
2022-09-27T16:13:49.5615331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:49.5616435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:49.5617031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:49.5617760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:49.6015835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:49.6016371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:49.6017691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:49.6018154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:49.8485697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:49.8493405Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-09-27T16:13:49.8494120Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-09-27T16:13:49.8589020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rf8k6ty
2022-09-27T16:13:49.8591970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rf8k6ty/_remote_module_non_scriptable.py
2022-09-27T16:13:49.8833758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:49.8843537Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-09-27T16:13:49.8844251Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-09-27T16:13:49.8945607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpre4u1tc4
2022-09-27T16:13:49.8948797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpre4u1tc4/_remote_module_non_scriptable.py
2022-09-27T16:13:50.3062443Z ok (4.041s)
2022-09-27T16:13:50.3062843Z 
2022-09-27T16:13:50.3063241Z ----------------------------------------------------------------------
2022-09-27T16:13:50.3063559Z Ran 1 test in 4.041s
2022-09-27T16:13:50.3063735Z 
2022-09-27T16:13:50.3063839Z OK
2022-09-27T16:13:50.3063938Z 
2022-09-27T16:13:50.3065483Z Generating XML reports...
2022-09-27T16:13:50.3103556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161346.xml
2022-09-27T16:13:52.4339120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:52.4339658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:52.4340240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:52.4340768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:52.7223563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:13:52.7238673Z 
2022-09-27T16:13:52.7238958Z Running tests...
2022-09-27T16:13:52.7239451Z ----------------------------------------------------------------------
2022-09-27T16:13:54.3599671Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:54.3792140Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28104
2022-09-27T16:13:54.3799438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28105
2022-09-27T16:13:56.0412553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:56.0413383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:56.0414645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:56.0415215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:56.0556322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:56.0556805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:56.0560152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:56.0560648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:56.3361713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:56.3487473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:57.7302152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo71cg2j
2022-09-27T16:13:57.7302785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo71cg2j/_remote_module_non_scriptable.py
2022-09-27T16:13:57.7595071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9h9x7w2
2022-09-27T16:13:57.7598357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9h9x7w2/_remote_module_non_scriptable.py
2022-09-27T16:13:59.4917044Z ok (6.767s)
2022-09-27T16:13:59.4917505Z 
2022-09-27T16:13:59.4918190Z ----------------------------------------------------------------------
2022-09-27T16:13:59.4918724Z Ran 1 test in 6.768s
2022-09-27T16:13:59.4919001Z 
2022-09-27T16:13:59.4919198Z OK
2022-09-27T16:13:59.4919837Z 
2022-09-27T16:13:59.4920120Z Generating XML reports...
2022-09-27T16:13:59.4964852Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161352.xml
2022-09-27T16:14:01.6034364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:01.6034929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:01.6035879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:01.6036407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:01.9005043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:01.9019708Z 
2022-09-27T16:14:01.9020023Z Running tests...
2022-09-27T16:14:01.9020487Z ----------------------------------------------------------------------
2022-09-27T16:14:03.5728474Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:03.5929354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28225
2022-09-27T16:14:03.5936080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28226
2022-09-27T16:14:05.2801575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:05.2802116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:05.2802730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:05.2803162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:05.2937372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:05.2937868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:05.2941078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:05.2941840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:05.5653671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:05.5681416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:06.9639669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eugggtd
2022-09-27T16:14:06.9640274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eugggtd/_remote_module_non_scriptable.py
2022-09-27T16:14:06.9767775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz191ihiw
2022-09-27T16:14:06.9770511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz191ihiw/_remote_module_non_scriptable.py
2022-09-27T16:14:08.7053109Z ok (6.803s)
2022-09-27T16:14:08.7053443Z 
2022-09-27T16:14:08.7053893Z ----------------------------------------------------------------------
2022-09-27T16:14:08.7054288Z Ran 1 test in 6.803s
2022-09-27T16:14:08.7054494Z 
2022-09-27T16:14:08.7054665Z OK
2022-09-27T16:14:08.7054908Z 
2022-09-27T16:14:08.7055042Z Generating XML reports...
2022-09-27T16:14:08.7093528Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161401.xml
2022-09-27T16:14:10.8271815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:10.8272375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:10.8273208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:10.8274052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:11.1200692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:11.1216682Z 
2022-09-27T16:14:11.1217141Z Running tests...
2022-09-27T16:14:11.1217686Z ----------------------------------------------------------------------
2022-09-27T16:14:12.7435298Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:12.7634118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28346
2022-09-27T16:14:12.7641315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28347
2022-09-27T16:14:14.4614679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:14.4615644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:14.4616837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:14.4617756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:14.4765304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:14.4766254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:14.4768777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:14.4770068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:14.7586829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:16.1604650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:16.1605871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzpuksib
2022-09-27T16:14:16.1606947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzpuksib/_remote_module_non_scriptable.py
2022-09-27T16:14:16.1744472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifr039yj
2022-09-27T16:14:16.1745875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifr039yj/_remote_module_non_scriptable.py
2022-09-27T16:14:17.7596219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:17.7597331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:17.8510463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:17.8511558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:18.4776932Z ok (7.356s)
2022-09-27T16:14:18.4777260Z 
2022-09-27T16:14:18.4777653Z ----------------------------------------------------------------------
2022-09-27T16:14:18.4777994Z Ran 1 test in 7.356s
2022-09-27T16:14:18.4778170Z 
2022-09-27T16:14:18.4778268Z OK
2022-09-27T16:14:18.4778386Z 
2022-09-27T16:14:18.4778526Z Generating XML reports...
2022-09-27T16:14:18.4817615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161411.xml
2022-09-27T16:14:20.6069763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:20.6070839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:20.6071752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:20.6072247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:20.8872975Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:20.8886239Z 
2022-09-27T16:14:20.8886693Z Running tests...
2022-09-27T16:14:20.8887987Z ----------------------------------------------------------------------
2022-09-27T16:14:22.4825156Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:22.4995690Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-09-27T16:14:22.4996301Z 
2022-09-27T16:14:22.4996599Z ----------------------------------------------------------------------
2022-09-27T16:14:22.4996929Z Ran 1 test in 1.611s
2022-09-27T16:14:22.4997100Z 
2022-09-27T16:14:22.4997213Z OK (skipped=1)
2022-09-27T16:14:22.4997374Z 
2022-09-27T16:14:22.4997506Z Generating XML reports...
2022-09-27T16:14:22.5031893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161420.xml
2022-09-27T16:14:24.5479468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:24.5480049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:24.5480648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:24.5481140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:24.8273020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:24.8287398Z 
2022-09-27T16:14:24.8287647Z Running tests...
2022-09-27T16:14:24.8288062Z ----------------------------------------------------------------------
2022-09-27T16:14:26.4226903Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:26.4388333Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2022-09-27T16:14:26.4389262Z 
2022-09-27T16:14:26.4389575Z ----------------------------------------------------------------------
2022-09-27T16:14:26.4389907Z Ran 1 test in 1.610s
2022-09-27T16:14:26.4390090Z 
2022-09-27T16:14:26.4390202Z OK (skipped=1)
2022-09-27T16:14:26.4390385Z 
2022-09-27T16:14:26.4390508Z Generating XML reports...
2022-09-27T16:14:26.4423511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161424.xml
2022-09-27T16:14:28.4971247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:28.4971805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:28.4972463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:28.4972954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:28.7732848Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:28.7747210Z 
2022-09-27T16:14:28.7747666Z Running tests...
2022-09-27T16:14:28.7748191Z ----------------------------------------------------------------------
2022-09-27T16:14:30.3759547Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:30.3925823Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2022-09-27T16:14:30.3926862Z 
2022-09-27T16:14:30.3927188Z ----------------------------------------------------------------------
2022-09-27T16:14:30.3927530Z Ran 1 test in 1.618s
2022-09-27T16:14:30.3927699Z 
2022-09-27T16:14:30.3927816Z OK (skipped=1)
2022-09-27T16:14:30.3927975Z 
2022-09-27T16:14:30.3928105Z Generating XML reports...
2022-09-27T16:14:30.3968786Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161428.xml
2022-09-27T16:14:32.4760206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.4761272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.4761897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.4762367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.7664081Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:32.7678459Z 
2022-09-27T16:14:32.7678846Z Running tests...
2022-09-27T16:14:32.7679386Z ----------------------------------------------------------------------
2022-09-27T16:14:34.4144547Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:34.4312935Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.663s)
2022-09-27T16:14:34.4313800Z 
2022-09-27T16:14:34.4314108Z ----------------------------------------------------------------------
2022-09-27T16:14:34.4314433Z Ran 1 test in 1.663s
2022-09-27T16:14:34.4314601Z 
2022-09-27T16:14:34.4314718Z OK (skipped=1)
2022-09-27T16:14:34.4314903Z 
2022-09-27T16:14:34.4315040Z Generating XML reports...
2022-09-27T16:14:34.4349201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161432.xml
2022-09-27T16:14:36.5142575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:36.5143119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:36.5143727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:36.5144217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:36.8055120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:36.8069492Z 
2022-09-27T16:14:36.8069800Z Running tests...
2022-09-27T16:14:36.8070228Z ----------------------------------------------------------------------
2022-09-27T16:14:38.4400225Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:38.4569014Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-09-27T16:14:38.4569952Z 
2022-09-27T16:14:38.4570224Z ----------------------------------------------------------------------
2022-09-27T16:14:38.4570579Z Ran 1 test in 1.650s
2022-09-27T16:14:38.4570751Z 
2022-09-27T16:14:38.4570867Z OK (skipped=1)
2022-09-27T16:14:38.4571033Z 
2022-09-27T16:14:38.4571171Z Generating XML reports...
2022-09-27T16:14:38.4605231Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161436.xml
2022-09-27T16:14:40.5359736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:40.5360763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:40.5362038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:40.5363004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:40.8287634Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:40.8303854Z 
2022-09-27T16:14:40.8304397Z Running tests...
2022-09-27T16:14:40.8304915Z ----------------------------------------------------------------------
2022-09-27T16:14:42.4668459Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:42.4841227Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-09-27T16:14:42.4842457Z 
2022-09-27T16:14:42.4842991Z ----------------------------------------------------------------------
2022-09-27T16:14:42.4843647Z Ran 1 test in 1.654s
2022-09-27T16:14:42.4843964Z 
2022-09-27T16:14:42.4844177Z OK (skipped=1)
2022-09-27T16:14:42.4844458Z 
2022-09-27T16:14:42.4844698Z Generating XML reports...
2022-09-27T16:14:42.4879986Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161440.xml
2022-09-27T16:14:44.5293152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.5294196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.5295447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.5296427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.8068917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:44.8084676Z 
2022-09-27T16:14:44.8085192Z Running tests...
2022-09-27T16:14:44.8085681Z ----------------------------------------------------------------------
2022-09-27T16:14:46.4387551Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:46.4579374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28682
2022-09-27T16:14:46.4586892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28683
2022-09-27T16:14:48.1491387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.1492008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.1492582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.1493062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.1496001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:48.1496475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:48.1499253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:48.1499799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:48.4225866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:48.4360726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:49.8400181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnr8a1wd
2022-09-27T16:14:49.8400918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnr8a1wd/_remote_module_non_scriptable.py
2022-09-27T16:14:49.8547264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_crq783
2022-09-27T16:14:49.8549959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_crq783/_remote_module_non_scriptable.py
2022-09-27T16:14:52.0708191Z ok (7.262s)
2022-09-27T16:14:52.0708432Z 
2022-09-27T16:14:52.0708815Z ----------------------------------------------------------------------
2022-09-27T16:14:52.0709159Z Ran 1 test in 7.262s
2022-09-27T16:14:52.0709315Z 
2022-09-27T16:14:52.0709414Z OK
2022-09-27T16:14:52.0709565Z 
2022-09-27T16:14:52.0709707Z Generating XML reports...
2022-09-27T16:14:52.0748419Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161444.xml
2022-09-27T16:14:54.1878767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:54.1879334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:54.1879921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:54.1880383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:54.4803000Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:14:54.4818311Z 
2022-09-27T16:14:54.4818418Z Running tests...
2022-09-27T16:14:54.4819311Z ----------------------------------------------------------------------
2022-09-27T16:14:56.1215263Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:56.1408451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28803
2022-09-27T16:14:56.1415796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28804
2022-09-27T16:14:57.8101256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:57.8102222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:57.8102807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:57.8103279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:57.8123262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:57.8123751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:57.8126509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:57.8127012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:58.0956349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:58.0957695Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:14:58.0990038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:58.0994128Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:14:59.5041426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpfadqjc
2022-09-27T16:14:59.5042283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpfadqjc/_remote_module_non_scriptable.py
2022-09-27T16:14:59.5567313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzo2j6u1
2022-09-27T16:14:59.5568266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzo2j6u1/_remote_module_non_scriptable.py
2022-09-27T16:15:01.2534802Z ok (6.771s)
2022-09-27T16:15:01.2535034Z 
2022-09-27T16:15:01.2535452Z ----------------------------------------------------------------------
2022-09-27T16:15:01.2535815Z Ran 1 test in 6.772s
2022-09-27T16:15:01.2535988Z 
2022-09-27T16:15:01.2536086Z OK
2022-09-27T16:15:01.2536222Z 
2022-09-27T16:15:01.2536363Z Generating XML reports...
2022-09-27T16:15:01.2574536Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161454.xml
2022-09-27T16:15:03.3982400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.3982997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.3983863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.3984316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.6901933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:03.6917382Z 
2022-09-27T16:15:03.6917579Z Running tests...
2022-09-27T16:15:03.6918015Z ----------------------------------------------------------------------
2022-09-27T16:15:05.3270164Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:05.3464150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28924
2022-09-27T16:15:05.3471458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28925
2022-09-27T16:15:07.0173223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.0174184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.0175382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.0176330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.0396380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:07.0397264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:07.0399680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:07.0400588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:07.3077264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:07.3078966Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:07.3456379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:07.3458762Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:08.7213740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_rwkq24
2022-09-27T16:15:08.7214916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_rwkq24/_remote_module_non_scriptable.py
2022-09-27T16:15:08.7700269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt_rau3f
2022-09-27T16:15:08.7702620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt_rau3f/_remote_module_non_scriptable.py
2022-09-27T16:15:10.5589221Z ok (6.867s)
2022-09-27T16:15:10.5589524Z 
2022-09-27T16:15:10.5589910Z ----------------------------------------------------------------------
2022-09-27T16:15:10.5590215Z Ran 1 test in 6.867s
2022-09-27T16:15:10.5590387Z 
2022-09-27T16:15:10.5590530Z OK
2022-09-27T16:15:10.5590640Z 
2022-09-27T16:15:10.5590773Z Generating XML reports...
2022-09-27T16:15:10.5630601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161503.xml
2022-09-27T16:15:12.6726513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.6727176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.6727979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.6728510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.9691383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:12.9706298Z 
2022-09-27T16:15:12.9706504Z Running tests...
2022-09-27T16:15:12.9706968Z ----------------------------------------------------------------------
2022-09-27T16:15:14.6266991Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:14.6469372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29045
2022-09-27T16:15:14.6475577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29046
2022-09-27T16:15:16.3109864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:16.3110399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:16.3110990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:16.3111476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:16.3324332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:16.3324795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:16.3327701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:16.3328203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:16.6111714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:16.6246684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:18.0383295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xeopwm3
2022-09-27T16:15:18.0384415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xeopwm3/_remote_module_non_scriptable.py
2022-09-27T16:15:18.0908763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl8qrlyq
2022-09-27T16:15:18.0909916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl8qrlyq/_remote_module_non_scriptable.py
2022-09-27T16:15:20.2605815Z ok (7.290s)
2022-09-27T16:15:20.2606100Z 
2022-09-27T16:15:20.2606824Z ----------------------------------------------------------------------
2022-09-27T16:15:20.2607225Z Ran 1 test in 7.290s
2022-09-27T16:15:20.2607393Z 
2022-09-27T16:15:20.2607491Z OK
2022-09-27T16:15:20.2607632Z 
2022-09-27T16:15:20.2607774Z Generating XML reports...
2022-09-27T16:15:20.2646514Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161512.xml
2022-09-27T16:15:22.3793895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:22.3794431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:22.3794999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:22.3795519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:22.6642870Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:22.6658469Z 
2022-09-27T16:15:22.6658957Z Running tests...
2022-09-27T16:15:22.6659493Z ----------------------------------------------------------------------
2022-09-27T16:15:24.2658290Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:24.2851701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29166
2022-09-27T16:15:24.2858831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29167
2022-09-27T16:15:25.9874732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:25.9875534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:25.9876159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:25.9876631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:26.0065478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:26.0066543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:26.0068476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:26.0069270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:26.2865215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:26.2990021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:27.7430406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5tbevr1
2022-09-27T16:15:27.7431625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5tbevr1/_remote_module_non_scriptable.py
2022-09-27T16:15:27.7543231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjpvux43
2022-09-27T16:15:27.7545950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjpvux43/_remote_module_non_scriptable.py
2022-09-27T16:15:30.2295693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.2296331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.2598299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.2599362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.2993928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.2995361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3313128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3314438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3626519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3627575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3949803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.3950865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4264732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4265774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4581548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4582611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4898016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.4899045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.5229951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.5230811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.5559400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.5560200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6043798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6044405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6355939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6356876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6668706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6669840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6982573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.6983587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7306431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7307440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7622985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7624017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7947165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:30.7948216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:31.7024017Z ok (9.036s)
2022-09-27T16:15:31.7024461Z 
2022-09-27T16:15:31.7026240Z ----------------------------------------------------------------------
2022-09-27T16:15:31.7026671Z Ran 1 test in 9.036s
2022-09-27T16:15:31.7026864Z 
2022-09-27T16:15:31.7026951Z OK
2022-09-27T16:15:31.7027086Z 
2022-09-27T16:15:31.7027231Z Generating XML reports...
2022-09-27T16:15:31.7063773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161522.xml
2022-09-27T16:15:33.8080831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:33.8081662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:33.8082277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:33.8082759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:34.0856383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:34.0869896Z 
2022-09-27T16:15:34.0870217Z Running tests...
2022-09-27T16:15:34.0870675Z ----------------------------------------------------------------------
2022-09-27T16:15:35.6816789Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:35.7008665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29287
2022-09-27T16:15:35.7015518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29288
2022-09-27T16:15:37.3801128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.3801940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.3802586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.3803055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.4059043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:37.4059528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:37.4062184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:37.4062654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:37.6763496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:37.6791759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:38.1074172Z skip: Need at least 4 CUDA devices (4.020s)
2022-09-27T16:15:38.1074676Z 
2022-09-27T16:15:38.1075099Z ----------------------------------------------------------------------
2022-09-27T16:15:38.1075473Z Ran 1 test in 4.020s
2022-09-27T16:15:38.1075660Z 
2022-09-27T16:15:38.1075787Z OK (skipped=1)
2022-09-27T16:15:38.1075929Z 
2022-09-27T16:15:38.1076086Z Generating XML reports...
2022-09-27T16:15:38.1114832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161534.xml
2022-09-27T16:15:40.1760270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:40.1761349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:40.1762592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:40.1763571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:40.4573545Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:40.4588217Z 
2022-09-27T16:15:40.4588481Z Running tests...
2022-09-27T16:15:40.4588938Z ----------------------------------------------------------------------
2022-09-27T16:15:42.0535495Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:42.0729017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29390
2022-09-27T16:15:42.0735419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29391
2022-09-27T16:15:43.7435374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:43.7436243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:43.7436886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:43.7437388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:43.7545068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:43.7545540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:43.7548494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:43.7549338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:44.0318996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:44.0323412Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0324598Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0325659Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0326698Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0328081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0329151Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0363374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:44.0370479Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0371707Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0372983Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0374008Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0375086Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.0376165Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:15:44.4798687Z ok (4.021s)
2022-09-27T16:15:44.4798924Z 
2022-09-27T16:15:44.4799404Z ----------------------------------------------------------------------
2022-09-27T16:15:44.4799754Z Ran 1 test in 4.021s
2022-09-27T16:15:44.4799902Z 
2022-09-27T16:15:44.4799983Z OK
2022-09-27T16:15:44.4800145Z 
2022-09-27T16:15:44.4800272Z Generating XML reports...
2022-09-27T16:15:44.4840497Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161540.xml
2022-09-27T16:15:46.5867532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:46.5868071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:46.5868645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:46.5869504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:46.8787122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:46.8801353Z 
2022-09-27T16:15:46.8801654Z Running tests...
2022-09-27T16:15:46.8802115Z ----------------------------------------------------------------------
2022-09-27T16:15:48.5425012Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:48.5624990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29493
2022-09-27T16:15:48.5631401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29494
2022-09-27T16:15:50.1650946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:50.1651551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:50.1652189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:50.1652685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:50.2334333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:50.2334832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:50.2336859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:50.2337361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:50.4294500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:50.5089593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:51.8138414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cozj71n
2022-09-27T16:15:51.8139026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cozj71n/_remote_module_non_scriptable.py
2022-09-27T16:15:51.9284045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpsul7_4
2022-09-27T16:15:51.9284638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpsul7_4/_remote_module_non_scriptable.py
2022-09-27T16:15:54.0760079Z ok (7.195s)
2022-09-27T16:15:54.0760361Z 
2022-09-27T16:15:54.0760790Z ----------------------------------------------------------------------
2022-09-27T16:15:54.0761112Z Ran 1 test in 7.196s
2022-09-27T16:15:54.0761290Z 
2022-09-27T16:15:54.0761392Z OK
2022-09-27T16:15:54.0761535Z 
2022-09-27T16:15:54.0761676Z Generating XML reports...
2022-09-27T16:15:54.0799179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161546.xml
2022-09-27T16:15:56.1281110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:56.1282157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:56.1283375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:56.1284321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:56.4167434Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:15:56.4183615Z 
2022-09-27T16:15:56.4184124Z Running tests...
2022-09-27T16:15:56.4184651Z ----------------------------------------------------------------------
2022-09-27T16:15:58.0400570Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:58.0601785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29614
2022-09-27T16:15:58.0608814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29615
2022-09-27T16:15:59.7475371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.7476430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.7478780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.7479734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:59.7606285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.7606786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.7610369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.7611107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:00.0457092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:00.0457597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:01.4938879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffqpfx1u
2022-09-27T16:16:01.4939531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6j9wzuqn
2022-09-27T16:16:01.4939992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffqpfx1u/_remote_module_non_scriptable.py
2022-09-27T16:16:01.4940571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6j9wzuqn/_remote_module_non_scriptable.py
2022-09-27T16:16:03.7735812Z ok (7.355s)
2022-09-27T16:16:03.7736382Z 
2022-09-27T16:16:03.7736785Z ----------------------------------------------------------------------
2022-09-27T16:16:03.7737197Z Ran 1 test in 7.355s
2022-09-27T16:16:03.7737350Z 
2022-09-27T16:16:03.7737452Z OK
2022-09-27T16:16:03.7737593Z 
2022-09-27T16:16:03.7737738Z Generating XML reports...
2022-09-27T16:16:03.7775235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161556.xml
2022-09-27T16:16:05.8788300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:05.8788790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:05.8789906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:05.8790445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:06.1607659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:06.1622456Z 
2022-09-27T16:16:06.1622781Z Running tests...
2022-09-27T16:16:06.1623296Z ----------------------------------------------------------------------
2022-09-27T16:16:07.7568158Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:07.7761461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29735
2022-09-27T16:16:07.7768096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29736
2022-09-27T16:16:09.4339048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:09.4339569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:09.4340173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:09.4340689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:09.4467720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:09.4468598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:09.4471095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:09.4471501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:09.7266366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:09.7334448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:11.1709960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlkzl636
2022-09-27T16:16:11.1710696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlkzl636/_remote_module_non_scriptable.py
2022-09-27T16:16:11.1898460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvb0v66a
2022-09-27T16:16:11.1901774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvb0v66a/_remote_module_non_scriptable.py
2022-09-27T16:16:12.8314321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:12.8314876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:13.4917817Z ok (7.329s)
2022-09-27T16:16:13.4918070Z 
2022-09-27T16:16:13.4918580Z ----------------------------------------------------------------------
2022-09-27T16:16:13.4919217Z Ran 1 test in 7.329s
2022-09-27T16:16:13.4919578Z 
2022-09-27T16:16:13.4919767Z OK
2022-09-27T16:16:13.4920058Z 
2022-09-27T16:16:13.4920332Z Generating XML reports...
2022-09-27T16:16:13.4955155Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161606.xml
2022-09-27T16:16:15.6018163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:15.6018719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:15.6019338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:15.6019753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:15.8936657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:15.8951488Z 
2022-09-27T16:16:15.8951654Z Running tests...
2022-09-27T16:16:15.8952209Z ----------------------------------------------------------------------
2022-09-27T16:16:17.5173969Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:17.5375659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29856
2022-09-27T16:16:17.5382288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29857
2022-09-27T16:16:19.1778738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:19.1779287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:19.1780539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:19.1781078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:19.2017864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:19.2018341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:19.2021396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:19.2022157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:19.4701223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:19.4839186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:20.8710764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapp1_ulu
2022-09-27T16:16:20.8711391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapp1_ulu/_remote_module_non_scriptable.py
2022-09-27T16:16:20.9140356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1c58mvy
2022-09-27T16:16:20.9142492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1c58mvy/_remote_module_non_scriptable.py
2022-09-27T16:16:22.5205083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:22.5225300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:23.1504669Z ok (7.255s)
2022-09-27T16:16:23.1504915Z 
2022-09-27T16:16:23.1505412Z ----------------------------------------------------------------------
2022-09-27T16:16:23.1505748Z Ran 1 test in 7.255s
2022-09-27T16:16:23.1505920Z 
2022-09-27T16:16:23.1506023Z OK
2022-09-27T16:16:23.1506194Z 
2022-09-27T16:16:23.1506318Z Generating XML reports...
2022-09-27T16:16:23.1544925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161615.xml
2022-09-27T16:16:25.2339529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:25.2340077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:25.2340677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:25.2341512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:25.5131352Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:25.5145242Z 
2022-09-27T16:16:25.5145623Z Running tests...
2022-09-27T16:16:25.5146144Z ----------------------------------------------------------------------
2022-09-27T16:16:27.1188776Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:27.1382333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29977
2022-09-27T16:16:27.1389101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29978
2022-09-27T16:16:28.7837479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:28.7838035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:28.7838887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:28.7839385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:28.8137049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:28.8137532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:28.8140537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:28.8141024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:29.0748782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:29.0951694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:29.5451020Z skip: Need at least 4 CUDA devices (4.030s)
2022-09-27T16:16:29.5451284Z 
2022-09-27T16:16:29.5451737Z ----------------------------------------------------------------------
2022-09-27T16:16:29.5452077Z Ran 1 test in 4.030s
2022-09-27T16:16:29.5452519Z 
2022-09-27T16:16:29.5452642Z OK (skipped=1)
2022-09-27T16:16:29.5452821Z 
2022-09-27T16:16:29.5452963Z Generating XML reports...
2022-09-27T16:16:29.5491179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161625.xml
2022-09-27T16:16:31.6197699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:31.6198202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:31.6198848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:31.6199315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:31.8978386Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:31.8992457Z 
2022-09-27T16:16:31.8992712Z Running tests...
2022-09-27T16:16:31.8993187Z ----------------------------------------------------------------------
2022-09-27T16:16:33.4984134Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:33.5177630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30080
2022-09-27T16:16:33.5184105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30081
2022-09-27T16:16:35.1798129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:35.1798659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:35.1799203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:35.1800017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:35.2059469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:35.2059998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:35.2063582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:35.2064081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:35.4710905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:35.4888431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:35.9244845Z skip: Need at least 8 CUDA devices (4.025s)
2022-09-27T16:16:35.9245178Z 
2022-09-27T16:16:35.9245625Z ----------------------------------------------------------------------
2022-09-27T16:16:35.9245987Z Ran 1 test in 4.025s
2022-09-27T16:16:35.9246183Z 
2022-09-27T16:16:35.9246303Z OK (skipped=1)
2022-09-27T16:16:35.9246471Z 
2022-09-27T16:16:35.9246579Z Generating XML reports...
2022-09-27T16:16:35.9284857Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161631.xml
2022-09-27T16:16:37.9985196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:37.9985715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:37.9986262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:37.9986730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:38.2789123Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:38.2803289Z 
2022-09-27T16:16:38.2803719Z Running tests...
2022-09-27T16:16:38.2804120Z ----------------------------------------------------------------------
2022-09-27T16:16:39.9016110Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:39.9207181Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30183
2022-09-27T16:16:39.9214699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30184
2022-09-27T16:16:41.5555405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:41.5556418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:41.5557571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:41.5558525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:41.5919790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:41.5920867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:41.5922402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:41.5923392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:41.8426042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:41.8882181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:43.8306460Z ok (5.550s)
2022-09-27T16:16:43.8306839Z 
2022-09-27T16:16:43.8307252Z ----------------------------------------------------------------------
2022-09-27T16:16:43.8307730Z Ran 1 test in 5.550s
2022-09-27T16:16:43.8307991Z 
2022-09-27T16:16:43.8308099Z OK
2022-09-27T16:16:43.8308249Z 
2022-09-27T16:16:43.8308399Z Generating XML reports...
2022-09-27T16:16:43.8346812Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161638.xml
2022-09-27T16:16:45.9213238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.9213784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.9215587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.9216097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:46.2126984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:46.2142549Z 
2022-09-27T16:16:46.2142850Z Running tests...
2022-09-27T16:16:46.2143303Z ----------------------------------------------------------------------
2022-09-27T16:16:47.8425277Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:47.8625445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30292
2022-09-27T16:16:47.8632094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30293
2022-09-27T16:16:49.5243455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:49.5244555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:49.5245764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:49.5246655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:49.6477538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:49.6478510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:49.6479654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:49.6481021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:49.7847463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:49.9226789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:50.2694979Z skip: Need at least 4 CUDA devices (4.055s)
2022-09-27T16:16:50.2695284Z 
2022-09-27T16:16:50.2695695Z ----------------------------------------------------------------------
2022-09-27T16:16:50.2696008Z Ran 1 test in 4.055s
2022-09-27T16:16:50.2696140Z 
2022-09-27T16:16:50.2696258Z OK (skipped=1)
2022-09-27T16:16:50.2696423Z 
2022-09-27T16:16:50.2696557Z Generating XML reports...
2022-09-27T16:16:50.2734714Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161646.xml
2022-09-27T16:16:52.3841839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:52.3842377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:52.3843003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:52.3843495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.6787189Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:16:52.6802010Z 
2022-09-27T16:16:52.6802174Z Running tests...
2022-09-27T16:16:52.6802629Z ----------------------------------------------------------------------
2022-09-27T16:16:54.3230094Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:54.3422297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30395
2022-09-27T16:16:54.3428862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30396
2022-09-27T16:16:56.0286814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:56.0287317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:56.0287929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:56.0288429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:56.0427769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:56.0428233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:56.0431545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:56.0432037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:56.3212313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:56.3354455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:57.7569370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi3nb2vp
2022-09-27T16:16:57.7570335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi3nb2vp/_remote_module_non_scriptable.py
2022-09-27T16:16:57.7921484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfh5p_2ha
2022-09-27T16:16:57.7922644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfh5p_2ha/_remote_module_non_scriptable.py
2022-09-27T16:16:59.3638919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:59.3639522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:16:59.9581515Z ok (7.278s)
2022-09-27T16:16:59.9581714Z 
2022-09-27T16:16:59.9582410Z ----------------------------------------------------------------------
2022-09-27T16:16:59.9582779Z Ran 1 test in 7.278s
2022-09-27T16:16:59.9582952Z 
2022-09-27T16:16:59.9583054Z OK
2022-09-27T16:16:59.9583201Z 
2022-09-27T16:16:59.9583336Z Generating XML reports...
2022-09-27T16:16:59.9621342Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161652.xml
2022-09-27T16:17:02.0904539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.0905185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.0905764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.0906298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.3731913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:17:02.3746777Z 
2022-09-27T16:17:02.3747148Z Running tests...
2022-09-27T16:17:02.3747491Z ----------------------------------------------------------------------
2022-09-27T16:17:03.9703394Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:03.9897114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30516
2022-09-27T16:17:03.9903659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30517
2022-09-27T16:17:05.6760327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:05.6760927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:05.6761862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:05.6762306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:05.7489660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:05.7490317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:05.7493012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:05.7493464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:05.9546880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:06.0257728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:07.3356526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bihdnd5
2022-09-27T16:17:07.3357246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bihdnd5/_remote_module_non_scriptable.py
2022-09-27T16:17:07.4164306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkc60nyr
2022-09-27T16:17:07.4164973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkc60nyr/_remote_module_non_scriptable.py
2022-09-27T16:17:09.0493533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:17:09.0513540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:17:09.7034924Z ok (7.328s)
2022-09-27T16:17:09.7035220Z 
2022-09-27T16:17:09.7035657Z ----------------------------------------------------------------------
2022-09-27T16:17:09.7036023Z Ran 1 test in 7.329s
2022-09-27T16:17:09.7036196Z 
2022-09-27T16:17:09.7036301Z OK
2022-09-27T16:17:09.7036425Z 
2022-09-27T16:17:09.7036572Z Generating XML reports...
2022-09-27T16:17:09.7074179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161702.xml
2022-09-27T16:17:11.8282304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:11.8282843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:11.8283466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:11.8283940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:12.1072365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:17:12.1086843Z 
2022-09-27T16:17:12.1086999Z Running tests...
2022-09-27T16:17:12.1087790Z ----------------------------------------------------------------------
2022-09-27T16:17:13.7209293Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:13.7402239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30637
2022-09-27T16:17:13.7408616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30638
2022-09-27T16:17:15.3840818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:15.3841378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:15.3841967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:15.3842461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:15.4094530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:15.4095008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:15.4098009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:15.4098534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:15.6745047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:15.6874278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:34.5819528Z ok (22.473s)
2022-09-27T16:17:34.5819774Z 
2022-09-27T16:17:34.5822001Z ----------------------------------------------------------------------
2022-09-27T16:17:34.5822376Z Ran 1 test in 22.473s
2022-09-27T16:17:34.5822573Z 
2022-09-27T16:17:34.5822681Z OK
2022-09-27T16:17:34.5822827Z 
2022-09-27T16:17:34.5825374Z Generating XML reports...
2022-09-27T16:17:34.5860169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161712.xml
2022-09-27T16:17:36.6934229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:36.6934758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:36.6935770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:36.6936272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:36.9844369Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:17:36.9859343Z 
2022-09-27T16:17:36.9859628Z Running tests...
2022-09-27T16:17:36.9860099Z ----------------------------------------------------------------------
2022-09-27T16:17:36.9874842Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-09-27T16:17:38.6053139Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:38.6245323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30758
2022-09-27T16:17:38.6253264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30759
2022-09-27T16:17:40.2594017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.2594542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.2595138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.2595631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.2952473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.2952943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.2955693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.2956186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.5448707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:40.5892852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:41.9312681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbogdr3rs
2022-09-27T16:17:41.9313315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbogdr3rs/_remote_module_non_scriptable.py
2022-09-27T16:17:42.0174753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1sgsj97
2022-09-27T16:17:42.0175568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1sgsj97/_remote_module_non_scriptable.py
2022-09-27T16:17:44.1379817Z ok (7.152s)
2022-09-27T16:17:44.1380206Z 
2022-09-27T16:17:44.1381178Z ----------------------------------------------------------------------
2022-09-27T16:17:44.1381773Z Ran 1 test in 7.152s
2022-09-27T16:17:44.1382077Z 
2022-09-27T16:17:44.1382239Z OK
2022-09-27T16:17:44.1382492Z 
2022-09-27T16:17:44.1382765Z Generating XML reports...
2022-09-27T16:17:44.1421183Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161736.xml
2022-09-27T16:17:46.2055193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:46.2056247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:46.2057429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:46.2058395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:46.5020973Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:17:46.5036932Z 
2022-09-27T16:17:46.5037465Z Running tests...
2022-09-27T16:17:46.5037982Z ----------------------------------------------------------------------
2022-09-27T16:17:48.1322405Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:48.1525792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30875
2022-09-27T16:17:48.1532274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30876
2022-09-27T16:17:49.7474202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:49.7475230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:49.7476372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:49.7477310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:49.8263319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:49.8264152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:49.8265603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:49.8266389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:50.0174195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:50.1039758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:51.4119311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdss99wo
2022-09-27T16:17:51.4120496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdss99wo/_remote_module_non_scriptable.py
2022-09-27T16:17:51.5207884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro9pz2_z
2022-09-27T16:17:51.5209000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro9pz2_z/_remote_module_non_scriptable.py
2022-09-27T16:17:53.1668445Z ok (6.663s)
2022-09-27T16:17:53.1668715Z 
2022-09-27T16:17:53.1669130Z ----------------------------------------------------------------------
2022-09-27T16:17:53.1669486Z Ran 1 test in 6.663s
2022-09-27T16:17:53.1669670Z 
2022-09-27T16:17:53.1669748Z OK
2022-09-27T16:17:53.1670556Z 
2022-09-27T16:17:53.1670703Z Generating XML reports...
2022-09-27T16:17:53.1708247Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161746.xml
2022-09-27T16:17:55.2680804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:55.2681653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:55.2682537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:55.2683050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:55.5560007Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:17:55.5574448Z 
2022-09-27T16:17:55.5574896Z Running tests...
2022-09-27T16:17:55.5575429Z ----------------------------------------------------------------------
2022-09-27T16:17:57.1820121Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:57.2011193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30992
2022-09-27T16:17:57.2018118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30993
2022-09-27T16:17:58.8596724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:58.8597296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:58.8597912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:58.8598425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:58.8875672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:58.8876448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:58.8878122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:58.8878619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:59.1520953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:59.1524616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:59.1661605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:59.1667432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:59.1668572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:59.1729005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:59.6077909Z ok (4.050s)
2022-09-27T16:17:59.6078181Z 
2022-09-27T16:17:59.6078597Z ----------------------------------------------------------------------
2022-09-27T16:17:59.6078963Z Ran 1 test in 4.050s
2022-09-27T16:17:59.6079131Z 
2022-09-27T16:17:59.6079204Z OK
2022-09-27T16:17:59.6079350Z 
2022-09-27T16:17:59.6079468Z Generating XML reports...
2022-09-27T16:17:59.6116786Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161755.xml
2022-09-27T16:18:01.7344964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:01.7345536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:01.7350367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:01.7350901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:02.0336168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:02.0351013Z 
2022-09-27T16:18:02.0351263Z Running tests...
2022-09-27T16:18:02.0351673Z ----------------------------------------------------------------------
2022-09-27T16:18:03.6726637Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:03.6927886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31099
2022-09-27T16:18:03.6934453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31100
2022-09-27T16:18:05.3665557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:05.3666102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:05.3666683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:05.3667168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:05.4025744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:05.4026252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:05.4028653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:05.4029175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:05.6566059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:05.6567443Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:05.7006300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:05.7007844Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:07.0407079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjwhr_y8
2022-09-27T16:18:07.0408116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjwhr_y8/_remote_module_non_scriptable.py
2022-09-27T16:18:07.1351792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54bqet92
2022-09-27T16:18:07.1352860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54bqet92/_remote_module_non_scriptable.py
2022-09-27T16:18:08.2014224Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2015379Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2064107Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2065146Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2114208Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2115308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2163641Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2164678Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2213519Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2214586Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2263368Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2264420Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:08.2313110Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.2314166Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:08.8048132Z ok (6.769s)
2022-09-27T16:18:08.8048399Z 
2022-09-27T16:18:08.8048827Z ----------------------------------------------------------------------
2022-09-27T16:18:08.8049150Z Ran 1 test in 6.770s
2022-09-27T16:18:08.8049286Z 
2022-09-27T16:18:08.8049379Z OK
2022-09-27T16:18:08.8049516Z 
2022-09-27T16:18:08.8050058Z Generating XML reports...
2022-09-27T16:18:08.8088799Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161802.xml
2022-09-27T16:18:10.8715300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:10.8715990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:10.8716613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:10.8717047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:11.1520424Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:11.1535475Z 
2022-09-27T16:18:11.1535902Z Running tests...
2022-09-27T16:18:11.1536443Z ----------------------------------------------------------------------
2022-09-27T16:18:12.7471799Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:12.7665430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31220
2022-09-27T16:18:12.7671715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31221
2022-09-27T16:18:14.4034608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:14.4035599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:14.4036805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:14.4037769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:14.4405972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:14.4406872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:14.4408752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:14.4410448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:14.6933161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:14.6934853Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:14.7386968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:14.7388885Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:16.0718132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dcfc03c
2022-09-27T16:18:16.0719216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dcfc03c/_remote_module_non_scriptable.py
2022-09-27T16:18:16.1488217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllx1zzy0
2022-09-27T16:18:16.1489195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllx1zzy0/_remote_module_non_scriptable.py
2022-09-27T16:18:17.3287817Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3290720Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3338205Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3340422Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3389573Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3391727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3440858Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3443314Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3492426Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3494568Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3543726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3546073Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2022-09-27T16:18:17.3594512Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.3596699Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:18:17.9788727Z ok (6.825s)
2022-09-27T16:18:17.9789133Z 
2022-09-27T16:18:17.9789954Z ----------------------------------------------------------------------
2022-09-27T16:18:17.9790314Z Ran 1 test in 6.825s
2022-09-27T16:18:17.9790488Z 
2022-09-27T16:18:17.9790605Z OK
2022-09-27T16:18:17.9790756Z 
2022-09-27T16:18:17.9790896Z Generating XML reports...
2022-09-27T16:18:17.9830800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161811.xml
2022-09-27T16:18:20.1051639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:20.1052716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:20.1053444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:20.1053929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:20.3857166Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:20.3870609Z 
2022-09-27T16:18:20.3871182Z Running tests...
2022-09-27T16:18:20.3871802Z ----------------------------------------------------------------------
2022-09-27T16:18:21.9826744Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:22.0019248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31341
2022-09-27T16:18:22.0025688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31342
2022-09-27T16:18:23.6327940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:23.6328998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:23.6330161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:23.6330675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:23.6642856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:23.6643620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:23.6645427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:23.6646237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:23.9204133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:23.9653359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:25.3010181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejrd1ltf
2022-09-27T16:18:25.3011173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejrd1ltf/_remote_module_non_scriptable.py
2022-09-27T16:18:25.3584209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyekdwfez
2022-09-27T16:18:25.3586082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyekdwfez/_remote_module_non_scriptable.py
2022-09-27T16:18:27.8586361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:18:27.8586931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:18:28.7176293Z ok (8.330s)
2022-09-27T16:18:28.7176544Z 
2022-09-27T16:18:28.7176938Z ----------------------------------------------------------------------
2022-09-27T16:18:28.7177297Z Ran 1 test in 8.330s
2022-09-27T16:18:28.7177474Z 
2022-09-27T16:18:28.7177576Z OK
2022-09-27T16:18:28.7177717Z 
2022-09-27T16:18:28.7177857Z Generating XML reports...
2022-09-27T16:18:28.7216137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161820.xml
2022-09-27T16:18:30.8452650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:30.8453228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:30.8453853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:30.8454349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:31.1358193Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:31.1372315Z 
2022-09-27T16:18:31.1372534Z Running tests...
2022-09-27T16:18:31.1373000Z ----------------------------------------------------------------------
2022-09-27T16:18:32.7665884Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:32.7864994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31462
2022-09-27T16:18:32.7871558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31463
2022-09-27T16:18:34.4874794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:34.4875352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:34.4876306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:34.4876793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:34.4918958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:34.4919436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:34.4922076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:34.4922571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:34.7724226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:34.7728046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:36.1688807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwkdtagm
2022-09-27T16:18:36.1690144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwkdtagm/_remote_module_non_scriptable.py
2022-09-27T16:18:36.1948268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_87t5yz
2022-09-27T16:18:36.1949302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_87t5yz/_remote_module_non_scriptable.py
2022-09-27T16:18:37.9696971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:18:37.9697555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:18:38.6003838Z ok (7.463s)
2022-09-27T16:18:38.6004247Z 
2022-09-27T16:18:38.6005306Z ----------------------------------------------------------------------
2022-09-27T16:18:38.6005904Z Ran 1 test in 7.463s
2022-09-27T16:18:38.6006191Z 
2022-09-27T16:18:38.6006343Z OK
2022-09-27T16:18:38.6006586Z 
2022-09-27T16:18:38.6006844Z Generating XML reports...
2022-09-27T16:18:38.6045339Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161831.xml
2022-09-27T16:18:40.6870368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:40.6870893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:40.6871738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:40.6872209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:40.9700470Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:40.9714237Z 
2022-09-27T16:18:40.9714684Z Running tests...
2022-09-27T16:18:40.9715270Z ----------------------------------------------------------------------
2022-09-27T16:18:42.5570036Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:42.5761303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31583
2022-09-27T16:18:42.5768385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31584
2022-09-27T16:18:42.5775205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31585
2022-09-27T16:18:44.2919851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:44.2920889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:44.2922033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:44.2923053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:44.2993332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:44.2994187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:44.2995813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:44.2996326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:44.3159318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:44.3159766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:44.3162178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:44.3162686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:44.5968305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:44.5968834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:44.5996862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:45.0841713Z skip: Need at least 3 CUDA devices (4.112s)
2022-09-27T16:18:45.0842290Z 
2022-09-27T16:18:45.0842693Z ----------------------------------------------------------------------
2022-09-27T16:18:45.0843025Z Ran 1 test in 4.113s
2022-09-27T16:18:45.0843200Z 
2022-09-27T16:18:45.0843323Z OK (skipped=1)
2022-09-27T16:18:45.0843515Z 
2022-09-27T16:18:45.0843668Z Generating XML reports...
2022-09-27T16:18:45.0881121Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161840.xml
2022-09-27T16:18:47.1582084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:47.1583106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:47.1584357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:47.1585344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:47.4486970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:47.4502777Z 
2022-09-27T16:18:47.4503342Z Running tests...
2022-09-27T16:18:47.4503847Z ----------------------------------------------------------------------
2022-09-27T16:18:49.0905538Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:49.1106089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31720
2022-09-27T16:18:49.1112518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31721
2022-09-27T16:18:49.1119515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31722
2022-09-27T16:18:50.7924947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:50.7925979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:50.7927209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:50.7928158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:50.7929360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:50.7930663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:50.7931846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:50.7932850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:50.8156853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:50.8158256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:50.8160851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:50.8161399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.0870699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:51.0921208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:51.1031575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:51.6188111Z skip: Need at least 3 CUDA devices (4.168s)
2022-09-27T16:18:51.6188432Z 
2022-09-27T16:18:51.6188865Z ----------------------------------------------------------------------
2022-09-27T16:18:51.6189198Z Ran 1 test in 4.168s
2022-09-27T16:18:51.6189382Z 
2022-09-27T16:18:51.6189500Z OK (skipped=1)
2022-09-27T16:18:51.6189663Z 
2022-09-27T16:18:51.6189797Z Generating XML reports...
2022-09-27T16:18:51.6227116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161847.xml
2022-09-27T16:18:53.7404781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:53.7405411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:53.7406036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:53.7406527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.0338952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:54.0352872Z 
2022-09-27T16:18:54.0353226Z Running tests...
2022-09-27T16:18:54.0353748Z ----------------------------------------------------------------------
2022-09-27T16:18:54.0359735Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.001s)
2022-09-27T16:18:54.0360107Z 
2022-09-27T16:18:54.0360404Z ----------------------------------------------------------------------
2022-09-27T16:18:54.0360744Z Ran 1 test in 0.001s
2022-09-27T16:18:54.0360914Z 
2022-09-27T16:18:54.0361031Z OK (skipped=1)
2022-09-27T16:18:54.0361169Z 
2022-09-27T16:18:54.0361304Z Generating XML reports...
2022-09-27T16:18:54.0396357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161854.xml
2022-09-27T16:18:55.9991300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:55.9991833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:55.9992451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:55.9992955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:56.2901214Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:18:56.2916040Z 
2022-09-27T16:18:56.2916355Z Running tests...
2022-09-27T16:18:56.2916820Z ----------------------------------------------------------------------
2022-09-27T16:18:57.9246420Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:57.9447504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31890
2022-09-27T16:18:57.9454586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31891
2022-09-27T16:18:57.9461732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31892
2022-09-27T16:18:59.6177537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:59.6178675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:59.6179326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:59.6180009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:59.6673928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:59.6674413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:59.6676896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:59.6677606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:59.6937783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:59.6938288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:59.6940855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:59.6941479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:59.9027812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:18:59.9427040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:59.9758620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:00.4531256Z skip: Need at least 3 CUDA devices (4.161s)
2022-09-27T16:19:00.4531539Z 
2022-09-27T16:19:00.4532260Z ----------------------------------------------------------------------
2022-09-27T16:19:00.4532637Z Ran 1 test in 4.161s
2022-09-27T16:19:00.4532824Z 
2022-09-27T16:19:00.4532917Z OK (skipped=1)
2022-09-27T16:19:00.4533079Z 
2022-09-27T16:19:00.4533239Z Generating XML reports...
2022-09-27T16:19:00.4572391Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161856.xml
2022-09-27T16:19:02.6109777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:02.6110363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:02.6110973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:02.6111460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:02.9021319Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:02.9035303Z 
2022-09-27T16:19:02.9035768Z Running tests...
2022-09-27T16:19:02.9036253Z ----------------------------------------------------------------------
2022-09-27T16:19:04.4917459Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:04.5111007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32027
2022-09-27T16:19:04.5116869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32028
2022-09-27T16:19:04.5123277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32029
2022-09-27T16:19:06.1863603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:06.1864102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:06.1864746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:06.1865205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:06.2244371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:06.2244870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:06.2247144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:06.2247618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:06.2845009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:06.2845574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:06.2846204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:06.2846739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:06.4807665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:06.4954848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:06.5668976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:19:07.0196052Z ok (4.116s)
2022-09-27T16:19:07.0196440Z 
2022-09-27T16:19:07.0196973Z ----------------------------------------------------------------------
2022-09-27T16:19:07.0197301Z Ran 1 test in 4.116s
2022-09-27T16:19:07.0197476Z 
2022-09-27T16:19:07.0197577Z OK
2022-09-27T16:19:07.0197695Z 
2022-09-27T16:19:07.0197841Z Generating XML reports...
2022-09-27T16:19:07.0235861Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161902.xml
2022-09-27T16:19:09.0947222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:09.0947842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:09.0948421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:09.0948883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:09.3735848Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:09.3749573Z 
2022-09-27T16:19:09.3749819Z Running tests...
2022-09-27T16:19:09.3750296Z ----------------------------------------------------------------------
2022-09-27T16:19:10.9682134Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:10.9875949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32164
2022-09-27T16:19:10.9883061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32165
2022-09-27T16:19:10.9889775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32166
2022-09-27T16:19:12.6096952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:12.6097957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:12.6099222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:12.6100113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:12.6683676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:12.6684673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:12.6685848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:12.6686760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:12.6878671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:12.6879666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:12.6880870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:12.6881807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:12.8799986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:12.9510476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:12.9735715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:19:13.3966436Z ok (4.021s)
2022-09-27T16:19:13.3966649Z 
2022-09-27T16:19:13.3967114Z ----------------------------------------------------------------------
2022-09-27T16:19:13.3967491Z Ran 1 test in 4.022s
2022-09-27T16:19:13.3967648Z 
2022-09-27T16:19:13.3967754Z OK
2022-09-27T16:19:13.3967898Z 
2022-09-27T16:19:13.3968014Z Generating XML reports...
2022-09-27T16:19:13.4004961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161909.xml
2022-09-27T16:19:15.5222967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:15.5223507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:15.5224128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:15.5224596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:15.8153664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:15.8167458Z 
2022-09-27T16:19:15.8167774Z Running tests...
2022-09-27T16:19:15.8168251Z ----------------------------------------------------------------------
2022-09-27T16:19:17.4679073Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:17.4871258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32301
2022-09-27T16:19:17.4878248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32302
2022-09-27T16:19:17.4884692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32303
2022-09-27T16:19:19.1395014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:19.1396058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:19.1397330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:19.1398329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:19.1684780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:19.1685711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:19.1688690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:19.1690032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:19.1966153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:19.1967091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:19.1970115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:19.1971061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:19.4378360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:19:19.4490558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:19.4816738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:19.9963253Z ok (4.179s)
2022-09-27T16:19:19.9963530Z 
2022-09-27T16:19:19.9963931Z ----------------------------------------------------------------------
2022-09-27T16:19:19.9964288Z Ran 1 test in 4.179s
2022-09-27T16:19:19.9964495Z 
2022-09-27T16:19:19.9964547Z OK
2022-09-27T16:19:19.9964666Z 
2022-09-27T16:19:19.9964795Z Generating XML reports...
2022-09-27T16:19:20.0002433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161915.xml
2022-09-27T16:19:22.1119184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:22.1119747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:22.1120359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:22.1120881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:22.3987860Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:22.4002910Z 
2022-09-27T16:19:22.4003189Z Running tests...
2022-09-27T16:19:22.4003644Z ----------------------------------------------------------------------
2022-09-27T16:19:22.4019159Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2022-09-27T16:19:22.4019465Z 
2022-09-27T16:19:22.4020081Z ----------------------------------------------------------------------
2022-09-27T16:19:22.4020482Z Ran 1 test in 0.002s
2022-09-27T16:19:22.4020595Z 
2022-09-27T16:19:22.4020794Z OK (skipped=1)
2022-09-27T16:19:22.4020909Z 
2022-09-27T16:19:22.4021043Z Generating XML reports...
2022-09-27T16:19:22.4062265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161922.xml
2022-09-27T16:19:24.3602413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:24.3603009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:24.3603826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:24.3604334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:24.6495725Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:24.6514653Z 
2022-09-27T16:19:24.6514918Z Running tests...
2022-09-27T16:19:24.6515366Z ----------------------------------------------------------------------
2022-09-27T16:19:26.2914561Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:26.3115192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32471
2022-09-27T16:19:26.3122416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32472
2022-09-27T16:19:26.3129870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32473
2022-09-27T16:19:28.0214844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:28.0215881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:28.0217047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:28.0217985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:28.0219165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:28.0220531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:28.0221743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:28.0222680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:28.0396220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:28.0396708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:28.0398747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:28.0399234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:28.3401679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:19:28.3402743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:28.3446582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:28.8207248Z skip: Need at least 3 CUDA devices (4.169s)
2022-09-27T16:19:28.8207504Z 
2022-09-27T16:19:28.8207909Z ----------------------------------------------------------------------
2022-09-27T16:19:28.8208266Z Ran 1 test in 4.169s
2022-09-27T16:19:28.8208432Z 
2022-09-27T16:19:28.8208546Z OK (skipped=1)
2022-09-27T16:19:28.8208702Z 
2022-09-27T16:19:28.8208845Z Generating XML reports...
2022-09-27T16:19:28.8247991Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161924.xml
2022-09-27T16:19:30.9680991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:30.9681628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:30.9682259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:30.9682743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:31.2497917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:31.2511303Z 
2022-09-27T16:19:31.2511527Z Running tests...
2022-09-27T16:19:31.2512422Z ----------------------------------------------------------------------
2022-09-27T16:19:31.2518766Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2022-09-27T16:19:31.2519350Z 
2022-09-27T16:19:31.2519932Z ----------------------------------------------------------------------
2022-09-27T16:19:31.2520616Z Ran 1 test in 0.001s
2022-09-27T16:19:31.2520845Z 
2022-09-27T16:19:31.2520962Z OK (skipped=1)
2022-09-27T16:19:31.2521123Z 
2022-09-27T16:19:31.2521258Z Generating XML reports...
2022-09-27T16:19:31.2554755Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220927161931.xml
2022-09-27T16:19:33.1833008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:33.1833586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:33.1834169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:33.1834637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:33.4792506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:33.4806647Z 
2022-09-27T16:19:33.4806776Z Running tests...
2022-09-27T16:19:33.4807758Z ----------------------------------------------------------------------
2022-09-27T16:19:35.1439269Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:35.1638378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32641
2022-09-27T16:19:35.1645445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32642
2022-09-27T16:19:36.8455944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:36.8456485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:36.8457391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:36.8457865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:36.8618031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:36.8618510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:36.8620726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:36.8621203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:37.1442614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:37.1444574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:37.1457757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:37.1462139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:37.1462975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:37.1547711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:39.0737228Z ok (5.593s)
2022-09-27T16:19:39.0737507Z 
2022-09-27T16:19:39.0737933Z ----------------------------------------------------------------------
2022-09-27T16:19:39.0738272Z Ran 1 test in 5.593s
2022-09-27T16:19:39.0738442Z 
2022-09-27T16:19:39.0738541Z OK
2022-09-27T16:19:39.0738659Z 
2022-09-27T16:19:39.0738822Z Generating XML reports...
2022-09-27T16:19:39.0776670Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161933.xml
2022-09-27T16:19:41.1887877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:41.1888452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:41.1888984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:41.1889474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:41.4651980Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:41.4665963Z 
2022-09-27T16:19:41.4666457Z Running tests...
2022-09-27T16:19:41.4666986Z ----------------------------------------------------------------------
2022-09-27T16:19:43.0678173Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:43.0870948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32750
2022-09-27T16:19:43.0877523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32751
2022-09-27T16:19:44.7187299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:44.7187810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:44.7188428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:44.7188910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:44.7560887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:44.7561322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:44.7564151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:44.7564642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:45.0065472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:45.0067905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:45.0371614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:45.0375711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:45.0376556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:45.0474451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:48.1994387Z ok (6.732s)
2022-09-27T16:19:48.1994862Z 
2022-09-27T16:19:48.1995519Z ----------------------------------------------------------------------
2022-09-27T16:19:48.1995882Z Ran 1 test in 6.733s
2022-09-27T16:19:48.1996057Z 
2022-09-27T16:19:48.1996157Z OK
2022-09-27T16:19:48.1996274Z 
2022-09-27T16:19:48.1996418Z Generating XML reports...
2022-09-27T16:19:48.2033973Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161941.xml
2022-09-27T16:19:50.2748133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:50.2748751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:50.2749353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:50.2749872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:50.5562729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:50.5576876Z 
2022-09-27T16:19:50.5577103Z Running tests...
2022-09-27T16:19:50.5577576Z ----------------------------------------------------------------------
2022-09-27T16:19:52.1653848Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:52.1846833Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32867
2022-09-27T16:19:52.1854207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32868
2022-09-27T16:19:53.8275953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:53.8276545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:53.8277136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:53.8277603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:53.8602726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:53.8603191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:53.8605841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:53.8606345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:54.1147851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:54.1149258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:54.1407823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:54.1411734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:54.1413177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:54.1454959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:57.2971548Z ok (6.739s)
2022-09-27T16:19:57.2971848Z 
2022-09-27T16:19:57.2972310Z ----------------------------------------------------------------------
2022-09-27T16:19:57.2972699Z Ran 1 test in 6.739s
2022-09-27T16:19:57.2972871Z 
2022-09-27T16:19:57.2972976Z OK
2022-09-27T16:19:57.2973120Z 
2022-09-27T16:19:57.2973251Z Generating XML reports...
2022-09-27T16:19:57.3010682Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161950.xml
2022-09-27T16:19:59.3994920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:59.3995457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:59.3996283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:59.3996753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:59.6784418Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:19:59.6797959Z 
2022-09-27T16:19:59.6798107Z Running tests...
2022-09-27T16:19:59.6799193Z ----------------------------------------------------------------------
2022-09-27T16:20:01.2827153Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:01.3020936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32984
2022-09-27T16:20:01.3027337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32985
2022-09-27T16:20:02.9859829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:02.9860367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:02.9865803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:02.9866748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:03.0101404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:03.0101918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:03.0103179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:03.0103676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:03.2841520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:03.2844118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:03.2957596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:03.2961802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:03.2963026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:03.3048883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:06.5148117Z ok (6.835s)
2022-09-27T16:20:06.5148508Z 
2022-09-27T16:20:06.5149179Z ----------------------------------------------------------------------
2022-09-27T16:20:06.5149806Z Ran 1 test in 6.835s
2022-09-27T16:20:06.5150089Z 
2022-09-27T16:20:06.5150259Z OK
2022-09-27T16:20:06.5150466Z 
2022-09-27T16:20:06.5150712Z Generating XML reports...
2022-09-27T16:20:06.5189285Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161959.xml
2022-09-27T16:20:08.6292891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:08.6293500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:08.6294149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:08.6294636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:08.9078681Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:08.9092922Z 
2022-09-27T16:20:08.9093401Z Running tests...
2022-09-27T16:20:08.9093898Z ----------------------------------------------------------------------
2022-09-27T16:20:10.5470977Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:10.5664984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33101
2022-09-27T16:20:10.5672406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33102
2022-09-27T16:20:12.2004441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:12.2005013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:12.2005954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:12.2006453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:12.2381703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:12.2382190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:12.2384675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:12.2385173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:12.4842192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:12.4845161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:12.5149872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:12.5153934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:12.5154797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:12.5251840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:15.6788898Z ok (6.769s)
2022-09-27T16:20:15.6789161Z 
2022-09-27T16:20:15.6789603Z ----------------------------------------------------------------------
2022-09-27T16:20:15.6789933Z Ran 1 test in 6.769s
2022-09-27T16:20:15.6790105Z 
2022-09-27T16:20:15.6790204Z OK
2022-09-27T16:20:15.6790344Z 
2022-09-27T16:20:15.6790485Z Generating XML reports...
2022-09-27T16:20:15.6828267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162008.xml
2022-09-27T16:20:17.8279468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:17.8280382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:17.8280979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:17.8281440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:18.1194585Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:18.1209871Z 
2022-09-27T16:20:18.1210368Z Running tests...
2022-09-27T16:20:18.1210901Z ----------------------------------------------------------------------
2022-09-27T16:20:19.7507569Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:19.7708018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33218
2022-09-27T16:20:19.7714142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33219
2022-09-27T16:20:21.4123341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:21.4123906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:21.4124495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:21.4124983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:21.4441654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:21.4442110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:21.4444796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:21.4445603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:21.7024084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:21.7025536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:21.7237619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:21.7241735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:21.7242568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:21.7331211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:24.9832486Z ok (6.862s)
2022-09-27T16:20:24.9832884Z 
2022-09-27T16:20:24.9833347Z ----------------------------------------------------------------------
2022-09-27T16:20:24.9833729Z Ran 1 test in 6.862s
2022-09-27T16:20:24.9833877Z 
2022-09-27T16:20:24.9833977Z OK
2022-09-27T16:20:24.9834118Z 
2022-09-27T16:20:24.9834272Z Generating XML reports...
2022-09-27T16:20:24.9874376Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162018.xml
2022-09-27T16:20:27.0995111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:27.0995681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:27.0996283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:27.0996726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:27.3784081Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:27.3797803Z 
2022-09-27T16:20:27.3798286Z Running tests...
2022-09-27T16:20:27.3798804Z ----------------------------------------------------------------------
2022-09-27T16:20:28.9819131Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:29.0011137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33335
2022-09-27T16:20:29.0017945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33336
2022-09-27T16:20:30.6761568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:30.6762584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:30.6763805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:30.6764719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:30.7107949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:30.7108983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:30.7110729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:30.7111727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:30.9710280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:30.9711409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:30.9916562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:30.9921384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:30.9923201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:31.0017501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:34.1137385Z ok (6.734s)
2022-09-27T16:20:34.1137656Z 
2022-09-27T16:20:34.1138082Z ----------------------------------------------------------------------
2022-09-27T16:20:34.1138441Z Ran 1 test in 6.734s
2022-09-27T16:20:34.1138618Z 
2022-09-27T16:20:34.1138720Z OK
2022-09-27T16:20:34.1139699Z 
2022-09-27T16:20:34.1139862Z Generating XML reports...
2022-09-27T16:20:34.1176238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162027.xml
2022-09-27T16:20:36.1772262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.1772852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.1773693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.1774087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.4553933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:36.4567547Z 
2022-09-27T16:20:36.4567815Z Running tests...
2022-09-27T16:20:36.4568225Z ----------------------------------------------------------------------
2022-09-27T16:20:38.0664326Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:38.0855223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33452
2022-09-27T16:20:38.0861123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33453
2022-09-27T16:20:39.7528166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:39.7528695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:39.7529299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:39.7530354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:39.7653270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:39.7653787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:39.7657111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:39.7657626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:40.0395627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:40.0397978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:40.0492889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:40.0497583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:40.0498537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:40.0500308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:41.9954433Z ok (5.538s)
2022-09-27T16:20:41.9954707Z 
2022-09-27T16:20:41.9955159Z ----------------------------------------------------------------------
2022-09-27T16:20:41.9955522Z Ran 1 test in 5.539s
2022-09-27T16:20:41.9955697Z 
2022-09-27T16:20:41.9955775Z OK
2022-09-27T16:20:41.9955916Z 
2022-09-27T16:20:41.9956068Z Generating XML reports...
2022-09-27T16:20:41.9994615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162036.xml
2022-09-27T16:20:44.1057184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:44.1057748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:44.1058330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:44.1058812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:44.3804487Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:44.3818809Z 
2022-09-27T16:20:44.3819006Z Running tests...
2022-09-27T16:20:44.3819461Z ----------------------------------------------------------------------
2022-09-27T16:20:45.9816260Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:46.0007328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33561
2022-09-27T16:20:46.0014431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33562
2022-09-27T16:20:47.6533566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:47.6534184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:47.6534785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:47.6535271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:47.6729805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:47.6730539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:47.6733503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:47.6734009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:47.9460993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:47.9463137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:47.9573282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:47.9577608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:47.9578434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:47.9667299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:51.1137478Z ok (6.731s)
2022-09-27T16:20:51.1137945Z 
2022-09-27T16:20:51.1138445Z ----------------------------------------------------------------------
2022-09-27T16:20:51.1138810Z Ran 1 test in 6.732s
2022-09-27T16:20:51.1138983Z 
2022-09-27T16:20:51.1139085Z OK
2022-09-27T16:20:51.1139219Z 
2022-09-27T16:20:51.1139358Z Generating XML reports...
2022-09-27T16:20:51.1234094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162044.xml
2022-09-27T16:20:53.1992904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:53.1993436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:53.1994027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:53.1994481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:53.4950802Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:20:53.4966033Z 
2022-09-27T16:20:53.4966496Z Running tests...
2022-09-27T16:20:53.4967279Z ----------------------------------------------------------------------
2022-09-27T16:20:55.1320122Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:55.1522074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33682
2022-09-27T16:20:55.1528566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33683
2022-09-27T16:20:56.8267357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:56.8268396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:56.8269614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:56.8270525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:56.8442889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:56.8443860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:56.8445993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:56.8446967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:57.1217101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:57.1218939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:57.1320989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:57.1325655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:57.1327124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:57.1424554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:04.1710850Z ok (10.674s)
2022-09-27T16:21:04.1711060Z 
2022-09-27T16:21:04.1711513Z ----------------------------------------------------------------------
2022-09-27T16:21:04.1711911Z Ran 1 test in 10.674s
2022-09-27T16:21:04.1712041Z 
2022-09-27T16:21:04.1712113Z OK
2022-09-27T16:21:04.1712306Z 
2022-09-27T16:21:04.1712450Z Generating XML reports...
2022-09-27T16:21:04.1750400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162053.xml
2022-09-27T16:21:06.2877775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:06.2878734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:06.2879356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:06.2879843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:06.5834015Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:06.5849503Z 
2022-09-27T16:21:06.5849725Z Running tests...
2022-09-27T16:21:06.5851115Z ----------------------------------------------------------------------
2022-09-27T16:21:08.2327975Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:08.2528098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33803
2022-09-27T16:21:08.2535258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33804
2022-09-27T16:21:09.9234542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:09.9235783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:09.9237131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:09.9237650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:09.9287149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:09.9287971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:09.9290882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:09.9291708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:10.2094555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:10.2096900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:10.2130917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:10.2135563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:10.2137107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:10.2200226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:13.4654059Z ok (6.880s)
2022-09-27T16:21:13.4654484Z 
2022-09-27T16:21:13.4655233Z ----------------------------------------------------------------------
2022-09-27T16:21:13.4655584Z Ran 1 test in 6.880s
2022-09-27T16:21:13.4655768Z 
2022-09-27T16:21:13.4655878Z OK
2022-09-27T16:21:13.4655999Z 
2022-09-27T16:21:13.4656118Z Generating XML reports...
2022-09-27T16:21:13.4695182Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162106.xml
2022-09-27T16:21:15.6131804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:15.6132737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:15.6133366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:15.6133888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:15.8994460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:15.9007995Z 
2022-09-27T16:21:15.9008265Z Running tests...
2022-09-27T16:21:15.9008706Z ----------------------------------------------------------------------
2022-09-27T16:21:17.4963777Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:17.5156269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33920
2022-09-27T16:21:17.5162821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33921
2022-09-27T16:21:19.1693985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:19.1694528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:19.1695184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:19.1695659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:19.1900917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:19.1901368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:19.1904558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:19.1905050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:19.4618689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:19.4620050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:19.4698144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:19.4702444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:19.4703712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:19.4723089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:21.4260220Z ok (5.525s)
2022-09-27T16:21:21.4260486Z 
2022-09-27T16:21:21.4260970Z ----------------------------------------------------------------------
2022-09-27T16:21:21.4261335Z Ran 1 test in 5.525s
2022-09-27T16:21:21.4261506Z 
2022-09-27T16:21:21.4261595Z OK
2022-09-27T16:21:21.4261740Z 
2022-09-27T16:21:21.4261880Z Generating XML reports...
2022-09-27T16:21:21.4299388Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162115.xml
2022-09-27T16:21:23.5333820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:23.5334352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:23.5334985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:23.5335449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:23.8086608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:23.8100261Z 
2022-09-27T16:21:23.8100399Z Running tests...
2022-09-27T16:21:23.8101057Z ----------------------------------------------------------------------
2022-09-27T16:21:25.4085770Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:25.4276781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34029
2022-09-27T16:21:25.4283293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34030
2022-09-27T16:21:27.1058465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:27.1058994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:27.1059598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:27.1060090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:27.1087722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:27.1088184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:27.1090751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:27.1091253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:27.3958433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:27.3960296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:27.3979281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:27.3983464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:27.3984822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:27.4063427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:30.5402896Z ok (6.730s)
2022-09-27T16:21:30.5403240Z 
2022-09-27T16:21:30.5403789Z ----------------------------------------------------------------------
2022-09-27T16:21:30.5404114Z Ran 1 test in 6.730s
2022-09-27T16:21:30.5404263Z 
2022-09-27T16:21:30.5404361Z OK
2022-09-27T16:21:30.5404500Z 
2022-09-27T16:21:30.5404638Z Generating XML reports...
2022-09-27T16:21:30.5442272Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162123.xml
2022-09-27T16:21:32.6232141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:32.6232749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:32.6233362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:32.6233873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:32.9083395Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:32.9097900Z 
2022-09-27T16:21:32.9098159Z Running tests...
2022-09-27T16:21:32.9098591Z ----------------------------------------------------------------------
2022-09-27T16:21:34.5114631Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:34.5294053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34146
2022-09-27T16:21:34.5300456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34147
2022-09-27T16:21:36.2037649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:36.2038233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:36.2038809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:36.2039650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:36.2525693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:36.2526217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:36.2528404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:36.2528904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:36.4842358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:36.4844226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:36.5276344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:36.5280666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:36.5281492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:36.5352739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:39.6418995Z ok (6.732s)
2022-09-27T16:21:39.6419262Z 
2022-09-27T16:21:39.6419693Z ----------------------------------------------------------------------
2022-09-27T16:21:39.6420074Z Ran 1 test in 6.732s
2022-09-27T16:21:39.6420222Z 
2022-09-27T16:21:39.6420326Z OK
2022-09-27T16:21:39.6420444Z 
2022-09-27T16:21:39.6420591Z Generating XML reports...
2022-09-27T16:21:39.6459273Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162132.xml
2022-09-27T16:21:41.7394110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:41.7394669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:41.7395292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:41.7395761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:42.0277162Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:42.0290513Z 
2022-09-27T16:21:42.0290867Z Running tests...
2022-09-27T16:21:42.0291296Z ----------------------------------------------------------------------
2022-09-27T16:21:43.6747019Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:43.6947127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34263
2022-09-27T16:21:43.6953453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34264
2022-09-27T16:21:45.3344775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:45.3345319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:45.3346265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:45.3346764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:45.3805918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:45.3806417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:45.3808554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:45.3809027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:45.6339777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:45.6342356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:45.6633112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:45.6636980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:45.6638448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:45.6647491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:47.6049010Z ok (5.575s)
2022-09-27T16:21:47.6049294Z 
2022-09-27T16:21:47.6049988Z ----------------------------------------------------------------------
2022-09-27T16:21:47.6050318Z Ran 1 test in 5.576s
2022-09-27T16:21:47.6050508Z 
2022-09-27T16:21:47.6050612Z OK
2022-09-27T16:21:47.6051922Z 
2022-09-27T16:21:47.6052247Z Generating XML reports...
2022-09-27T16:21:47.6104421Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162142.xml
2022-09-27T16:21:49.7247830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:49.7248328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:49.7248948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:49.7249375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:50.0035100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:50.0048623Z 
2022-09-27T16:21:50.0049115Z Running tests...
2022-09-27T16:21:50.0050117Z ----------------------------------------------------------------------
2022-09-27T16:21:51.6005151Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:51.6198354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34372
2022-09-27T16:21:51.6205133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34373
2022-09-27T16:21:53.3007439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:53.3008008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:53.3008619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:53.3009112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:53.3242882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:53.3243404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:53.3246767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:53.3247244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:53.5912415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:53.5914641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:53.5940645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:53.5945500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:53.5946654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:53.6017603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:56.8327700Z ok (6.828s)
2022-09-27T16:21:56.8327956Z 
2022-09-27T16:21:56.8328393Z ----------------------------------------------------------------------
2022-09-27T16:21:56.8328754Z Ran 1 test in 6.828s
2022-09-27T16:21:56.8328903Z 
2022-09-27T16:21:56.8329007Z OK
2022-09-27T16:21:56.8329147Z 
2022-09-27T16:21:56.8329290Z Generating XML reports...
2022-09-27T16:21:56.8368162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162149.xml
2022-09-27T16:21:58.9649374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:58.9650224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:58.9650845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:58.9651362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:59.2546808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:21:59.2561443Z 
2022-09-27T16:21:59.2561723Z Running tests...
2022-09-27T16:21:59.2562176Z ----------------------------------------------------------------------
2022-09-27T16:22:00.9092702Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:00.9291337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34493
2022-09-27T16:22:00.9297830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34494
2022-09-27T16:22:02.5973883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:02.5974499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:02.5975142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:02.5975657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:02.6210621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:02.6211123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:02.6214348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:02.6214848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:02.8949456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:02.8950961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:02.9100594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:02.9105260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:02.9106061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:02.9155491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:10.0481342Z ok (10.792s)
2022-09-27T16:22:10.0481606Z 
2022-09-27T16:22:10.0482050Z ----------------------------------------------------------------------
2022-09-27T16:22:10.0482403Z Ran 1 test in 10.792s
2022-09-27T16:22:10.0482578Z 
2022-09-27T16:22:10.0482670Z OK
2022-09-27T16:22:10.0482818Z 
2022-09-27T16:22:10.0482972Z Generating XML reports...
2022-09-27T16:22:10.0520805Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162159.xml
2022-09-27T16:22:12.1757704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:12.1758305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:12.1758912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:12.1759412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:12.4590394Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:22:12.4604115Z 
2022-09-27T16:22:12.4604414Z Running tests...
2022-09-27T16:22:12.4604883Z ----------------------------------------------------------------------
2022-09-27T16:22:14.0599985Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:14.0791263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34614
2022-09-27T16:22:14.0798549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34615
2022-09-27T16:22:15.7212242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:15.7212975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:15.7213647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:15.7214333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:15.7553307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:15.7553803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:15.7557264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:15.7557826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:16.0125919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:16.0128592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:16.0357507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:16.0361414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:16.0362594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:16.0433962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:18.0893677Z ok (5.629s)
2022-09-27T16:22:18.0894068Z 
2022-09-27T16:22:18.0894664Z ----------------------------------------------------------------------
2022-09-27T16:22:18.0895042Z Ran 1 test in 5.629s
2022-09-27T16:22:18.0895217Z 
2022-09-27T16:22:18.0895318Z OK
2022-09-27T16:22:18.0895461Z 
2022-09-27T16:22:18.0895607Z Generating XML reports...
2022-09-27T16:22:18.0934584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162212.xml
2022-09-27T16:22:20.2065866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:20.2066376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:20.2066973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:20.2067456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:20.5045397Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:22:20.5060856Z 
2022-09-27T16:22:20.5061194Z Running tests...
2022-09-27T16:22:20.5061954Z ----------------------------------------------------------------------
2022-09-27T16:22:22.1490348Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:22.1680603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:22.1681460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:22.1705453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:22.1706653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:22.1726233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:22.1727343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:22.1746378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:22.1747192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:22.1814875Z ok (1.675s)
2022-09-27T16:22:22.1816228Z 
2022-09-27T16:22:22.1816713Z ----------------------------------------------------------------------
2022-09-27T16:22:22.1817048Z Ran 1 test in 1.676s
2022-09-27T16:22:22.1817216Z 
2022-09-27T16:22:22.1817321Z OK
2022-09-27T16:22:22.1817460Z 
2022-09-27T16:22:22.1817595Z Generating XML reports...
2022-09-27T16:22:22.1851331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220927162220.xml
2022-09-27T16:22:24.2555747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:24.2556510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:24.2557158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:24.2557935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:24.5454064Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-09-27T16:22:24.5468865Z 
2022-09-27T16:22:24.5469199Z Running tests...
2022-09-27T16:22:24.5469861Z ----------------------------------------------------------------------
2022-09-27T16:22:26.1907142Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:26.2089087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:26.2090489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:28.2149820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:28.2150743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:22:29.2457430Z ok (4.698s)
2022-09-27T16:22:29.2457716Z 
2022-09-27T16:22:29.2458239Z ----------------------------------------------------------------------
2022-09-27T16:22:29.2458721Z Ran 1 test in 4.699s
2022-09-27T16:22:29.2458884Z 
2022-09-27T16:22:29.2458988Z OK
2022-09-27T16:22:29.2459108Z 
2022-09-27T16:22:29.2459250Z Generating XML reports...
2022-09-27T16:22:29.2499239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220927162224.xml
2022-09-27T16:22:29.9673331Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-09-27 16:22:29.966773]
2022-09-27T16:22:29.9674158Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:22:29.966877]
2022-09-27T16:22:31.9179204Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-09-27T16:22:31.9210281Z 
2022-09-27T16:22:31.9210666Z Running tests...
2022-09-27T16:22:31.9211096Z ----------------------------------------------------------------------
2022-09-27T16:22:33.5419728Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:33.5615164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34830
2022-09-27T16:22:33.5621187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34831
2022-09-27T16:22:35.2856297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:35.2857014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:35.2857710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:35.2858248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:35.2892338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:35.2892815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:35.2895834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:35.2896296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:35.5295664Z dist init r=1, world=2
2022-09-27T16:22:35.5300531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:35.5519390Z dist init r=0, world=2
2022-09-27T16:22:35.5525802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:35.5527436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:35.5606195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:37.0781095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:37.0782088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:37.0997630Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:22:37.0999139Z   warnings.warn(
2022-09-27T16:22:37.1001405Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:22:37.1002876Z   warnings.warn(
2022-09-27T16:22:37.5459221Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-09-27T16:22:37.5459725Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-09-27T16:22:37.5460091Z     (_fpw_module): Linear(in_features=2, out_features=1, bias=True)
2022-09-27T16:22:37.5460421Z   )
2022-09-27T16:22:37.5460651Z )
2022-09-27T16:22:37.5461000Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.BACKWARD_PRE
2022-09-27T16:22:37.5462334Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 222, in bad_backwards_hook
2022-09-27T16:22:37.5463130Z     with model.summon_full_params(model):
2022-09-27T16:22:37.5463739Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:37.5464326Z     return next(self.gen)
2022-09-27T16:22:37.5465501Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2887, in summon_full_params
2022-09-27T16:22:37.5466102Z     stack.enter_context(
2022-09-27T16:22:37.5466641Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-09-27T16:22:37.5467228Z     result = _cm_type.__enter__(cm)
2022-09-27T16:22:37.5467815Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:37.5468307Z     return next(self.gen)
2022-09-27T16:22:37.5469302Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2927, in _summon_full_params
2022-09-27T16:22:37.5470353Z     stack.enter_context(
2022-09-27T16:22:37.5471160Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-09-27T16:22:37.5471905Z     result = _cm_type.__enter__(cm)
2022-09-27T16:22:37.5472680Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:37.5473405Z     return next(self.gen)
2022-09-27T16:22:37.5474696Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2940, in _summon_full_params
2022-09-27T16:22:37.5475722Z     self._assert_state([TrainingState_.IDLE])
2022-09-27T16:22:37.5477366Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3549, in _assert_state
2022-09-27T16:22:37.5478354Z     traceback.print_stack()
2022-09-27T16:22:38.1730806Z ok (6.252s)
2022-09-27T16:22:38.1754124Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34915
2022-09-27T16:22:38.1760385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34916
2022-09-27T16:22:39.8643328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:39.8644579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:39.8645192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:39.8645689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:39.8946813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:39.8947308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:39.8950218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:39.8950731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:40.1214060Z dist init r=0, world=2
2022-09-27T16:22:40.1219064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:40.1394641Z dist init r=1, world=2
2022-09-27T16:22:40.1400319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:40.1401650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:40.1422983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:41.6326040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:41.6326598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:41.6341520Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:22:41.6342475Z   warnings.warn(
2022-09-27T16:22:41.6343884Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:22:41.6344826Z   warnings.warn(
2022-09-27T16:22:41.6582427Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-09-27T16:22:41.6582982Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-09-27T16:22:41.6583413Z     (_fpw_module): MyModule()
2022-09-27T16:22:41.6583678Z   )
2022-09-27T16:22:41.6583911Z )
2022-09-27T16:22:41.6584212Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.FORWARD
2022-09-27T16:22:41.6595586Z   File "<string>", line 1, in <module>
2022-09-27T16:22:41.6596134Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-09-27T16:22:41.6596873Z     exitcode = _main(fd, parent_sentinel)
2022-09-27T16:22:41.6597237Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-09-27T16:22:41.6597861Z     return self._bootstrap(parent_sentinel)
2022-09-27T16:22:41.6598465Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-09-27T16:22:41.6598847Z     self.run()
2022-09-27T16:22:41.6599170Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-09-27T16:22:41.6599548Z     self._target(*self._args, **self._kwargs)
2022-09-27T16:22:41.6600180Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 785, in _run
2022-09-27T16:22:41.6600511Z     self.run_test(test_name, pipe)
2022-09-27T16:22:41.6601069Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:22:41.6601464Z     getattr(self, test_name)()
2022-09-27T16:22:41.6601941Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:22:41.6602405Z     fn()
2022-09-27T16:22:41.6602901Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-09-27T16:22:41.6603310Z     return func(*args, **kwargs)
2022-09-27T16:22:41.6603718Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 213, in test_cannot_summon_full_params_from_forward
2022-09-27T16:22:41.6604112Z     model(model)
2022-09-27T16:22:41.6604553Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-09-27T16:22:41.6604977Z     return forward_call(*input, **kwargs)
2022-09-27T16:22:41.6605536Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2697, in forward
2022-09-27T16:22:41.6606011Z     output = self._fsdp_wrapped_module(*args, **kwargs)
2022-09-27T16:22:41.6606546Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-09-27T16:22:41.6606902Z     return forward_call(*input, **kwargs)
2022-09-27T16:22:41.6607547Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flatten_params_wrapper.py", line 165, in forward
2022-09-27T16:22:41.6608054Z     return self.module(*inputs, **kwinputs)
2022-09-27T16:22:41.6608520Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-09-27T16:22:41.6608899Z     return forward_call(*input, **kwargs)
2022-09-27T16:22:41.6609329Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 206, in forward
2022-09-27T16:22:41.6610217Z     with fsdp_module.summon_full_params(fsdp_module):
2022-09-27T16:22:41.6610586Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:41.6610969Z     return next(self.gen)
2022-09-27T16:22:41.6611511Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2887, in summon_full_params
2022-09-27T16:22:41.6611919Z     stack.enter_context(
2022-09-27T16:22:41.6612283Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-09-27T16:22:41.6612640Z     result = _cm_type.__enter__(cm)
2022-09-27T16:22:41.6612977Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:41.6613376Z     return next(self.gen)
2022-09-27T16:22:41.6613896Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2927, in _summon_full_params
2022-09-27T16:22:41.6614328Z     stack.enter_context(
2022-09-27T16:22:41.6614660Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-09-27T16:22:41.6615024Z     result = _cm_type.__enter__(cm)
2022-09-27T16:22:41.6615380Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-09-27T16:22:41.6615741Z     return next(self.gen)
2022-09-27T16:22:41.6616357Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2940, in _summon_full_params
2022-09-27T16:22:41.6616885Z     self._assert_state([TrainingState_.IDLE])
2022-09-27T16:22:41.6617420Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3549, in _assert_state
2022-09-27T16:22:41.6617826Z     traceback.print_stack()
2022-09-27T16:22:42.1851641Z ok (4.012s)
2022-09-27T16:22:42.1862915Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2022-09-27T16:22:42.1879029Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34996
2022-09-27T16:22:42.1885749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34997
2022-09-27T16:22:43.8431622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:43.8432239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:43.8432880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:43.8433359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:43.8744521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:43.8744990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:43.8748251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:43.8748754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:44.1002118Z dist init r=0, world=2
2022-09-27T16:22:44.1006125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:44.1233368Z dist init r=1, world=2
2022-09-27T16:22:44.1239512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:44.1240350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:44.1311871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:45.6393710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:45.6394264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:46.1975435Z ok (4.012s)
2022-09-27T16:22:46.1985749Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2022-09-27T16:22:46.2001695Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35077
2022-09-27T16:22:46.2007954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35078
2022-09-27T16:22:47.8832307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:47.8832903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:47.8833473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:47.8833965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:47.9133586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:47.9134073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:47.9137291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:47.9137765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:48.1407508Z dist init r=1, world=2
2022-09-27T16:22:48.1411213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:48.1617142Z dist init r=0, world=2
2022-09-27T16:22:48.1622887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:48.1624470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:48.1716909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:49.6888552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:49.6889115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:50.2097396Z ok (4.012s)
2022-09-27T16:22:50.2108334Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2022-09-27T16:22:50.2124392Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35158
2022-09-27T16:22:50.2131160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35159
2022-09-27T16:22:51.9020892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:51.9021436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:51.9022034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:51.9022528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:51.9182361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:51.9182862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:51.9185587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:51.9186356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:52.1645778Z dist init r=1, world=2
2022-09-27T16:22:52.1650175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:52.1711921Z dist init r=0, world=2
2022-09-27T16:22:52.1717814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:52.1718934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:52.1753041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:53.7026410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:53.7026983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:54.2222711Z ok (4.012s)
2022-09-27T16:22:54.2233643Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2022-09-27T16:22:54.2249422Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35239
2022-09-27T16:22:54.2256405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35240
2022-09-27T16:22:55.8968139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:55.8968676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:55.8969491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:55.8970407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:55.9158401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:55.9158898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:55.9162252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:55.9162715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:56.1594799Z dist init r=1, world=2
2022-09-27T16:22:56.1599247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:56.1684878Z dist init r=0, world=2
2022-09-27T16:22:56.1690745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:56.1692781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:56.1701893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:57.7055284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:58.3352208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:58.3352647Z ok (4.113s)
2022-09-27T16:22:58.3382338Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35320
2022-09-27T16:22:58.3388451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35321
2022-09-27T16:23:00.0129012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:00.0129904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:00.0130693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:00.0131456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:00.0506677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:00.0507137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:00.0510026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:00.0510538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:00.2754379Z dist init r=1, world=2
2022-09-27T16:23:00.2759189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:00.2980191Z dist init r=0, world=2
2022-09-27T16:23:00.2986470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:00.2987338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:00.3064486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:01.8261427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:01.8262027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:02.4480444Z ok (4.113s)
2022-09-27T16:23:02.4512512Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35401
2022-09-27T16:23:02.4518950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35402
2022-09-27T16:23:04.1811443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:04.1812057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:04.1812661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:04.1813162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:04.1992597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:04.1993083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:04.1996326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:04.1996846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:04.4452653Z dist init r=1, world=2
2022-09-27T16:23:04.4457252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:04.4555948Z dist init r=0, world=2
2022-09-27T16:23:04.4562323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:04.4563197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:04.4661661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:05.9713366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:05.9713918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:06.5611617Z ok (4.113s)
2022-09-27T16:23:06.5643012Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35482
2022-09-27T16:23:06.5649928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35483
2022-09-27T16:23:08.2638178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:08.2638765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:08.2639600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:08.2640095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:08.2666673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:08.2667139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:08.2670192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:08.2670689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:08.5167722Z dist init r=1, world=2
2022-09-27T16:23:08.5172033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:08.5229978Z dist init r=0, world=2
2022-09-27T16:23:08.5236141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:08.5237031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:08.5275364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:10.0629710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:10.0630610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:10.0859498Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:23:10.0860224Z   warnings.warn(
2022-09-27T16:23:10.0861190Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:23:10.0861872Z   warnings.warn(
2022-09-27T16:23:10.5740527Z ok (4.013s)
2022-09-27T16:23:10.5770816Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35563
2022-09-27T16:23:10.5777411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35564
2022-09-27T16:23:12.2825823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:12.2826392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:12.2827025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:12.2827515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:12.3173540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:12.3174060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:12.3177091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:12.3177889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:12.5404868Z dist init r=0, world=2
2022-09-27T16:23:12.5408961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:12.5638995Z dist init r=1, world=2
2022-09-27T16:23:12.5644296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:12.5645146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:12.5714278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:14.0961111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:14.0961671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:14.1179251Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:23:14.1180096Z   warnings.warn(
2022-09-27T16:23:14.1181101Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:23:14.1182079Z   warnings.warn(
2022-09-27T16:23:14.6869546Z ok (4.113s)
2022-09-27T16:23:14.6902283Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35644
2022-09-27T16:23:14.6908426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35645
2022-09-27T16:23:16.3838204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:16.3838735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:16.3839350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:16.3839817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:16.3973982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:16.3974479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:16.3977034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:16.3977560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:16.6380485Z dist init r=0, world=2
2022-09-27T16:23:16.6384799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:16.6486151Z dist init r=1, world=2
2022-09-27T16:23:16.6491999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:16.6493649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:16.6589257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:18.1770119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:18.1771621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:18.6999721Z ok (4.013s)
2022-09-27T16:23:18.7030350Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35725
2022-09-27T16:23:18.7036854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35726
2022-09-27T16:23:20.3717041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:20.3717578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:20.3718439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:20.3718932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:20.4555311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:20.4555880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:20.4556664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:20.4557123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:20.6254784Z dist init r=1, world=2
2022-09-27T16:23:20.6258835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:20.6965757Z dist init r=0, world=2
2022-09-27T16:23:20.6971361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:20.6972866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:20.7069835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:22.2124051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:22.8129201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:22.8129858Z ok (4.113s)
2022-09-27T16:23:22.8160120Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35806
2022-09-27T16:23:22.8166693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35807
2022-09-27T16:23:24.5155559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:24.5156101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:24.5156717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:24.5157192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:24.5566726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:24.5567207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:24.5570056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:24.5570735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:24.7713554Z dist init r=0, world=2
2022-09-27T16:23:24.7717329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:24.8006088Z dist init r=1, world=2
2022-09-27T16:23:24.8012003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:24.8013117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:24.8022366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:26.3283424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:26.8256742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:26.8257204Z ok (4.013s)
2022-09-27T16:23:26.8287424Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35887
2022-09-27T16:23:26.8294292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35888
2022-09-27T16:23:28.4797307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:28.4797902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:28.4798775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:28.4799228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:28.5156366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:28.5156793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:28.5159368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:28.5160106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:28.7344286Z dist init r=1, world=2
2022-09-27T16:23:28.7348996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:28.7626580Z dist init r=0, world=2
2022-09-27T16:23:28.7631988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:28.7633199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:28.7654636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:30.3076356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:30.3076888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:30.8382983Z ok (4.012s)
2022-09-27T16:23:30.8413603Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35968
2022-09-27T16:23:30.8420048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35969
2022-09-27T16:23:32.5048565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:32.5049136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:32.5049954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:32.5050439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:32.5459290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:32.5459802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:32.5462572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:32.5463336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:32.7575822Z dist init r=0, world=2
2022-09-27T16:23:32.7579838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:32.7907828Z dist init r=1, world=2
2022-09-27T16:23:32.7913524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:32.7914509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:32.7986358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:34.2918139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:34.2918714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:34.8511161Z ok (4.013s)
2022-09-27T16:23:34.8539948Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36049
2022-09-27T16:23:34.8546176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36050
2022-09-27T16:23:36.5086636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:36.5087208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:36.5087789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:36.5088283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:36.5518585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:36.5519099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:36.5521831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:36.5522333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:36.7613518Z dist init r=0, world=2
2022-09-27T16:23:36.7617488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:36.7949494Z dist init r=1, world=2
2022-09-27T16:23:36.7954963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:36.7955983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:36.8023871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:38.3135571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:38.3136162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:38.8635817Z ok (4.012s)
2022-09-27T16:23:38.8665520Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36130
2022-09-27T16:23:38.8671307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36131
2022-09-27T16:23:40.5618110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:40.5618651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:40.5619293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:40.5619762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:40.5629790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:40.5630265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:40.5633159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:40.5633628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:40.8093656Z dist init r=0, world=2
2022-09-27T16:23:40.8098013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:40.8192815Z dist init r=1, world=2
2022-09-27T16:23:40.8198086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:40.8199370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:40.8200631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:42.3439578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:42.3440117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:42.8759749Z ok (4.012s)
2022-09-27T16:23:42.8789323Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36211
2022-09-27T16:23:42.8795552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36212
2022-09-27T16:23:44.5443979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:44.5444575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:44.5445172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:44.5445672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:44.5771152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:44.5771648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:44.5774986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:44.5775492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:44.7979755Z dist init r=1, world=2
2022-09-27T16:23:44.7983935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:44.8218701Z dist init r=0, world=2
2022-09-27T16:23:44.8224636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:44.8225485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:44.8289258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:46.3371608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:46.3372459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:46.8882205Z ok (4.012s)
2022-09-27T16:23:46.8911678Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36292
2022-09-27T16:23:46.8917947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36293
2022-09-27T16:23:48.5851282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:48.5851883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:48.5852425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:48.5852915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:48.6266471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:48.6266967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:48.6269984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:48.6270489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:48.8415432Z dist init r=0, world=2
2022-09-27T16:23:48.8419404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:48.8710032Z dist init r=1, world=2
2022-09-27T16:23:48.8715638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:48.8716809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:48.8724595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:50.4208520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:50.4209005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:51.0008230Z ok (4.112s)
2022-09-27T16:23:51.0036926Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36373
2022-09-27T16:23:51.0043041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36374
2022-09-27T16:23:52.6703129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:52.6703700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:52.6704588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:52.6705082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:52.6955642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:52.6956120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:52.6959889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:52.6960374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:52.9259253Z dist init r=0, world=2
2022-09-27T16:23:52.9263885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:52.9411459Z dist init r=1, world=2
2022-09-27T16:23:52.9417573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:52.9418710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:52.9467968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:54.4551015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:54.4551590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:55.0132029Z ok (4.012s)
2022-09-27T16:23:55.0161912Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36454
2022-09-27T16:23:55.0168232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36455
2022-09-27T16:23:56.6837515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:56.6838022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:56.6839040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:56.6839613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:56.7174545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:56.7175057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:56.7178124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:56.7178629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:56.9474860Z dist init r=1, world=2
2022-09-27T16:23:56.9478972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:56.9561146Z dist init r=0, world=2
2022-09-27T16:23:56.9566981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:56.9568432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:56.9582004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:58.4673597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:58.4674171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:59.0260172Z ok (4.013s)
2022-09-27T16:23:59.0288872Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36535
2022-09-27T16:23:59.0295789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36536
2022-09-27T16:24:00.6900340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:00.6901416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:00.6902580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:00.6903565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:00.7297104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:00.7298107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:00.7300412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:00.7301384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:00.9482928Z dist init r=0, world=2
2022-09-27T16:24:00.9487818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:00.9769768Z dist init r=1, world=2
2022-09-27T16:24:00.9775742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:00.9776806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:00.9792794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:02.5188930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:02.5189892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:03.1386573Z ok (4.112s)
2022-09-27T16:24:03.1390793Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2022-09-27T16:24:03.1406544Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36616
2022-09-27T16:24:03.1413437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36617
2022-09-27T16:24:04.7573800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.7574985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.7576630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.7577176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:04.8257369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.8257927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.8258663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.8259586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:05.0073451Z dist init r=1, world=2
2022-09-27T16:24:05.0077799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:05.0626717Z dist init r=0, world=2
2022-09-27T16:24:05.0631973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:05.0632935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:05.0686680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:06.6035241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:06.6036246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:07.1518176Z ok (4.013s)
2022-09-27T16:24:07.1553324Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36697
2022-09-27T16:24:07.1559574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36698
2022-09-27T16:24:08.8239230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:08.8239780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:08.8240365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:08.8240852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:08.8485573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:08.8486047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:08.8489248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:08.8490349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:09.0818154Z dist init r=0, world=2
2022-09-27T16:24:09.0822374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:09.0949645Z dist init r=1, world=2
2022-09-27T16:24:09.0955675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:09.0956820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:09.1026242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:10.6128006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:10.6128553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:10.6354755Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:10.6355577Z   warnings.warn(
2022-09-27T16:24:10.6357043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:10.6357837Z   warnings.warn(
2022-09-27T16:24:11.6658065Z ok (4.514s)
2022-09-27T16:24:11.6692735Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36782
2022-09-27T16:24:11.6699192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36783
2022-09-27T16:24:13.3896430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:13.3896952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:13.3897964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:13.3898424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:13.4172236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:13.4172727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:13.4176373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:13.4176837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:13.6440374Z dist init r=0, world=2
2022-09-27T16:24:13.6444633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:13.6653849Z dist init r=1, world=2
2022-09-27T16:24:13.6659264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:13.6660514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:13.6750091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:15.1968065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:15.1968628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:15.2196087Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:15.2196981Z   warnings.warn(
2022-09-27T16:24:15.2198119Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:15.2198893Z   warnings.warn(
2022-09-27T16:24:16.1797771Z ok (4.514s)
2022-09-27T16:24:16.1834415Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36867
2022-09-27T16:24:16.1840829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36868
2022-09-27T16:24:17.8750550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:17.8751131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:17.8751987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:17.8752503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:17.8990002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:17.8990486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:17.8993369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:17.8993864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:18.1348388Z dist init r=1, world=2
2022-09-27T16:24:18.1352533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:18.1452078Z dist init r=0, world=2
2022-09-27T16:24:18.1458918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:18.1460469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:18.1557703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:19.6739048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:19.6739585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:19.6954903Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:19.6955723Z   warnings.warn(
2022-09-27T16:24:19.6989868Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:19.6990956Z   warnings.warn(
2022-09-27T16:24:20.1384645Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:24:20.1385453Z   warnings.warn(
2022-09-27T16:24:20.1387282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:24:20.1388160Z   warnings.warn(
2022-09-27T16:24:20.7946083Z ok (4.615s)
2022-09-27T16:24:20.7981542Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36952
2022-09-27T16:24:20.7988068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36953
2022-09-27T16:24:22.4771649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:22.4772508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:22.4773107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:22.4773611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:22.5391659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:22.5392127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:22.5392934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:22.5393397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:22.7263241Z dist init r=1, world=2
2022-09-27T16:24:22.7267808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:22.7798192Z dist init r=0, world=2
2022-09-27T16:24:22.7804017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:22.7804994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:22.7876601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:24.3027505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:24.3028095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:24.3236355Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:24.3237686Z   warnings.warn(
2022-09-27T16:24:24.3239091Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:24.3240082Z   warnings.warn(
2022-09-27T16:24:24.7934996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:24:24.7935757Z   warnings.warn(
2022-09-27T16:24:24.7938828Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:24:24.7939538Z   warnings.warn(
2022-09-27T16:24:25.3091458Z ok (4.514s)
2022-09-27T16:24:25.3126204Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37037
2022-09-27T16:24:25.3133091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37038
2022-09-27T16:24:27.0273258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:27.0273875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:27.0274480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:27.0274992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:27.0335339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:27.0335770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:27.0338792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:27.0339230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:27.2817037Z dist init r=0, world=2
2022-09-27T16:24:27.2821677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:27.2906448Z dist init r=1, world=2
2022-09-27T16:24:27.2911970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:27.2913171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:27.2924368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:28.8161446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:28.8162013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:28.8354302Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:28.8355392Z   warnings.warn(
2022-09-27T16:24:28.8356503Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:28.8357292Z   warnings.warn(
2022-09-27T16:24:29.8235644Z ok (4.514s)
2022-09-27T16:24:29.8271815Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37122
2022-09-27T16:24:29.8278017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37123
2022-09-27T16:24:31.5061218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:31.5062266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:31.5063412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:31.5064367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:31.5146023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:31.5146530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:31.5149795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:31.5150340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:31.7556680Z dist init r=1, world=2
2022-09-27T16:24:31.7560622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:31.7718906Z dist init r=0, world=2
2022-09-27T16:24:31.7725174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:31.7726714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:31.7765083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:33.3142196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:33.3143201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:33.3355358Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:33.3356207Z   warnings.warn(
2022-09-27T16:24:33.3391215Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:33.3392710Z   warnings.warn(
2022-09-27T16:24:34.3378128Z ok (4.514s)
2022-09-27T16:24:34.3413868Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37207
2022-09-27T16:24:34.3420254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37208
2022-09-27T16:24:36.0394645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:36.0395200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:36.0395813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:36.0396299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:36.0478374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:36.0478859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:36.0481722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:36.0482236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:36.2968823Z dist init r=1, world=2
2022-09-27T16:24:36.2973513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:36.2993589Z dist init r=0, world=2
2022-09-27T16:24:36.2999735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:36.3001223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:36.3077031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:37.8151535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:37.8354495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:37.8357086Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:37.8358642Z   warnings.warn(
2022-09-27T16:24:37.8360839Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:37.8362309Z   warnings.warn(
2022-09-27T16:24:38.8519447Z ok (4.514s)
2022-09-27T16:24:38.8556663Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37292
2022-09-27T16:24:38.8563044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37293
2022-09-27T16:24:40.5704877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:40.5705430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:40.5706056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:40.5706523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.6271699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:40.6272492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:40.6273828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:40.6274294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.8192065Z dist init r=1, world=2
2022-09-27T16:24:40.8195854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:40.8678830Z dist init r=0, world=2
2022-09-27T16:24:40.8684827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:40.8685969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:40.8703434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:42.4042646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:42.4043213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:42.4283604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:42.4284400Z   warnings.warn(
2022-09-27T16:24:42.4285821Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:42.4286593Z   warnings.warn(
2022-09-27T16:24:43.5665501Z ok (4.714s)
2022-09-27T16:24:43.5689474Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37377
2022-09-27T16:24:43.5696008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37378
2022-09-27T16:24:45.2482338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:45.2482943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:45.2483799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:45.2484323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:45.2847328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:45.2847821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:45.2851028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:45.2851781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:45.5068115Z dist init r=0, world=2
2022-09-27T16:24:45.5072094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:45.5303466Z dist init r=1, world=2
2022-09-27T16:24:45.5309195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:45.5310417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:45.5377264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:47.0487401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:47.5786324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:47.5786772Z ok (4.012s)
2022-09-27T16:24:47.5817328Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37458
2022-09-27T16:24:47.5823711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37459
2022-09-27T16:24:49.1953505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:49.1954485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:49.1955621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:49.1956552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:49.2629352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:49.2630252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:49.2631265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:49.2632064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:49.4484192Z dist init r=0, world=2
2022-09-27T16:24:49.4488285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:49.5415716Z dist init r=1, world=2
2022-09-27T16:24:49.5421248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:49.5422097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:49.5503242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:51.0934806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:51.0935752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:51.1155717Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:51.1157237Z   warnings.warn(
2022-09-27T16:24:51.1159403Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:51.1160881Z   warnings.warn(
2022-09-27T16:24:51.6931703Z ok (4.114s)
2022-09-27T16:24:51.6963653Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37539
2022-09-27T16:24:51.6970038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37540
2022-09-27T16:24:53.3085497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:53.3086571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:53.3087744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:53.3088697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:53.3417767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:53.3418745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:53.3420731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:53.3421634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:53.5690070Z dist init r=1, world=2
2022-09-27T16:24:53.5694774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:53.5770938Z dist init r=0, world=2
2022-09-27T16:24:53.5776440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:53.5777550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:53.5797704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:55.0947769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:55.0949516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:55.1157283Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:55.1158998Z   warnings.warn(
2022-09-27T16:24:55.1161301Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:55.1162689Z   warnings.warn(
2022-09-27T16:24:55.6071791Z ok (3.914s)
2022-09-27T16:24:55.6101754Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37620
2022-09-27T16:24:55.6107998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37621
2022-09-27T16:24:57.2204793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:57.2205414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:57.2206249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:57.2206741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:57.3352803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:57.3353380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:57.3354445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:57.3354913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:57.4694998Z dist init r=0, world=2
2022-09-27T16:24:57.4698901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:57.5630482Z dist init r=1, world=2
2022-09-27T16:24:57.5637061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:57.5638104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:57.5713231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:59.0716096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:59.0716694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:59.0914320Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:59.0915115Z   warnings.warn(
2022-09-27T16:24:59.0916585Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:24:59.0917381Z   warnings.warn(
2022-09-27T16:24:59.6199507Z ok (4.013s)
2022-09-27T16:24:59.6229969Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37701
2022-09-27T16:24:59.6236786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37702
2022-09-27T16:25:01.2249292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:01.2250647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:01.2251911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:01.2252796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:01.2686549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:01.2687543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:01.2689437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:01.2690825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:01.4782907Z dist init r=0, world=2
2022-09-27T16:25:01.4786907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:01.5172541Z dist init r=1, world=2
2022-09-27T16:25:01.5178925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:01.5180247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:01.5193699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:03.0548118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:03.0549096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:03.0756216Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:03.0757663Z   warnings.warn(
2022-09-27T16:25:03.0790901Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:03.0792421Z   warnings.warn(
2022-09-27T16:25:03.6347228Z ok (4.015s)
2022-09-27T16:25:03.6380314Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37782
2022-09-27T16:25:03.6386644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37783
2022-09-27T16:25:05.3031118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:05.3032467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:05.3033682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:05.3034627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:05.3311926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:05.3312783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:05.3315059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:05.3315886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:05.5607163Z dist init r=1, world=2
2022-09-27T16:25:05.5611936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:05.5670192Z dist init r=0, world=2
2022-09-27T16:25:05.5675996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:05.5677135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:05.5715141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:07.0792490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:07.0793414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:07.0995782Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:07.0997681Z   warnings.warn(
2022-09-27T16:25:07.0999892Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:07.1001361Z   warnings.warn(
2022-09-27T16:25:07.6477865Z ok (4.013s)
2022-09-27T16:25:07.6508668Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37863
2022-09-27T16:25:07.6515176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37864
2022-09-27T16:25:09.3553001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:09.3553517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:09.3554377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:09.3554858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:09.3906028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:09.3906516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:09.3909512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:09.3910189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:09.6105024Z dist init r=0, world=2
2022-09-27T16:25:09.6109217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:09.6336012Z dist init r=1, world=2
2022-09-27T16:25:09.6341649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:09.6342475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:09.6414996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:11.1552213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:11.1552768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:11.1757371Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:11.1758197Z   warnings.warn(
2022-09-27T16:25:11.1759335Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:11.1760083Z   warnings.warn(
2022-09-27T16:25:11.6607151Z ok (4.013s)
2022-09-27T16:25:11.6637836Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37944
2022-09-27T16:25:11.6644398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37945
2022-09-27T16:25:13.4062103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:13.4062638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:13.4063247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:13.4063712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:13.4080602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:13.4081079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:13.4084553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:13.4085034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:13.6437675Z dist init r=0, world=2
2022-09-27T16:25:13.6442302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:13.6565201Z dist init r=1, world=2
2022-09-27T16:25:13.6570386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:13.6571358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:13.6646262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:15.1487280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:15.1488194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:15.1717013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:15.1717954Z   warnings.warn(
2022-09-27T16:25:15.1719096Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:15.1719864Z   warnings.warn(
2022-09-27T16:25:15.7738400Z ok (4.113s)
2022-09-27T16:25:15.7769191Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38025
2022-09-27T16:25:15.7775862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38026
2022-09-27T16:25:17.4291943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:17.4292501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:17.4293089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:17.4293555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:17.4623734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:17.4624247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:17.4625730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:17.4626253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:17.6846909Z dist init r=0, world=2
2022-09-27T16:25:17.6851228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:17.7084968Z dist init r=1, world=2
2022-09-27T16:25:17.7090487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:17.7091846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:17.7156563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:19.2333243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:19.2333888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:19.2594126Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:19.2594928Z   warnings.warn(
2022-09-27T16:25:19.2596382Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:25:19.2597163Z   warnings.warn(
2022-09-27T16:25:19.7867765Z ok (4.013s)
2022-09-27T16:25:19.7895823Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38106
2022-09-27T16:25:19.7902085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38107
2022-09-27T16:25:21.5412454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:21.5413045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:21.5414433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:21.5414909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:21.5551443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:21.5551924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:21.5555029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:21.5555508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:21.8063679Z dist init r=0, world=2
2022-09-27T16:25:21.8067073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:21.8172658Z dist init r=1, world=2
2022-09-27T16:25:21.8173135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:21.8173926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:21.8174621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:23.3296842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:23.3297471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:23.8995591Z ok (4.113s)
2022-09-27T16:25:23.9022288Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38187
2022-09-27T16:25:23.9028602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38188
2022-09-27T16:25:25.5822637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:25.5823177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:25.5824006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:25.5824524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:25.6047182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:25.6047667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:25.6050936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:25.6051437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:25.8359416Z dist init r=0, world=2
2022-09-27T16:25:25.8363910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:25.8498196Z dist init r=1, world=2
2022-09-27T16:25:25.8503490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:25.8504324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:25.8567786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:27.3942994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:27.3943584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:27.9119748Z ok (4.012s)
2022-09-27T16:25:27.9143182Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38268
2022-09-27T16:25:27.9149266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38269
2022-09-27T16:25:29.6014868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:29.6015796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:29.6016430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:29.6016915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:29.6324907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:29.6325368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:29.6328524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:29.6329043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:29.8578132Z dist init r=0, world=2
2022-09-27T16:25:29.8582062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:29.8822321Z dist init r=1, world=2
2022-09-27T16:25:29.8827815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:29.8829018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:29.8887139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:31.4308478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:31.4309053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:32.0242555Z ok (4.112s)
2022-09-27T16:25:32.0267390Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38349
2022-09-27T16:25:32.0273956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38350
2022-09-27T16:25:33.7376567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:33.7377143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:33.7377761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:33.7378234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:33.7486431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:33.7487189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:33.7490152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:33.7490893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:33.9997372Z dist init r=1, world=2
2022-09-27T16:25:34.0001095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:34.0032876Z dist init r=0, world=2
2022-09-27T16:25:34.0038714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:34.0039723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:34.0103681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:35.5449017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:35.5449811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:36.1368503Z ok (4.112s)
2022-09-27T16:25:36.1390995Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38430
2022-09-27T16:25:36.1397462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38431
2022-09-27T16:25:37.8545277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:37.8545824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:37.8546472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:37.8546945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:37.8627310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:37.8628077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:37.8630921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:37.8631394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:38.1082308Z dist init r=0, world=2
2022-09-27T16:25:38.1086270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:38.1117982Z dist init r=1, world=2
2022-09-27T16:25:38.1123652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:38.1125012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:38.1189456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:39.6479491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:39.6480056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:40.2489406Z ok (4.112s)
2022-09-27T16:25:40.2513726Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38511
2022-09-27T16:25:40.2519841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38512
2022-09-27T16:25:41.9107037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:41.9107604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:41.9108230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:41.9108722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:41.9326797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:41.9327276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:41.9329983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:41.9330801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:42.1749633Z dist init r=1, world=2
2022-09-27T16:25:42.1753771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:42.1864949Z dist init r=0, world=2
2022-09-27T16:25:42.1870754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:42.1871897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:42.1958370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:43.7062026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:43.7062583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:44.2608223Z ok (4.012s)
2022-09-27T16:25:44.2631846Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38592
2022-09-27T16:25:44.2637449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38593
2022-09-27T16:25:45.9182456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:45.9183269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:45.9183902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:45.9184398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:45.9451906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:45.9452369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:45.9455289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:45.9455795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:46.1725323Z dist init r=0, world=2
2022-09-27T16:25:46.1730118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:46.1896113Z dist init r=1, world=2
2022-09-27T16:25:46.1902009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:46.1902835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:46.1934079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:47.7141689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:47.7142585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:48.2724609Z ok (4.012s)
2022-09-27T16:25:48.2747120Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38673
2022-09-27T16:25:48.2753436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38674
2022-09-27T16:25:49.9853925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:49.9854514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:49.9856364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:49.9856864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:50.0026030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:50.0026479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:50.0029365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:50.0029823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:50.2537190Z dist init r=0, world=2
2022-09-27T16:25:50.2541567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:50.2560297Z dist init r=1, world=2
2022-09-27T16:25:50.2565899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:50.2567135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:50.2644850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:51.7928265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:51.7929183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:52.3845449Z ok (4.112s)
2022-09-27T16:25:52.3868628Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38754
2022-09-27T16:25:52.3875533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38755
2022-09-27T16:25:54.0806476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:54.0807510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:54.0808705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:54.0810144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:54.0868288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:54.0868768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:54.0871573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:54.0872073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:54.3260431Z dist init r=0, world=2
2022-09-27T16:25:54.3264437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:54.3437315Z dist init r=1, world=2
2022-09-27T16:25:54.3443499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:54.3445351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:54.3468977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:55.8708618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:55.8709663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:56.3966782Z ok (4.012s)
2022-09-27T16:25:56.3989733Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38835
2022-09-27T16:25:56.3995939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38836
2022-09-27T16:25:58.0971467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:58.0972262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:58.0973146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:58.0973822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:58.1037666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:58.1038483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:58.1040201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:58.1040875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:58.3428680Z dist init r=0, world=2
2022-09-27T16:25:58.3432859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:58.3637449Z dist init r=1, world=2
2022-09-27T16:25:58.3644159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:58.3645736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:58.3738233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:59.8952179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:59.8953158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:00.5086592Z ok (4.112s)
2022-09-27T16:26:00.5110097Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38916
2022-09-27T16:26:00.5116033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38917
2022-09-27T16:26:02.2483142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:02.2483689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:02.2484589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:02.2485076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:02.2510974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:02.2511445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:02.2514990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:02.2515480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:02.4997762Z dist init r=0, world=2
2022-09-27T16:26:02.5001966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:02.5058712Z dist init r=1, world=2
2022-09-27T16:26:02.5064322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:02.5065176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:02.5104663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:04.0580351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:04.0581007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:04.6207800Z ok (4.112s)
2022-09-27T16:26:04.6231799Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38997
2022-09-27T16:26:04.6237937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38998
2022-09-27T16:26:06.3039877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:06.3040399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:06.3043289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:06.3043797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:06.3070080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:06.3070569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:06.3073238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:06.3073741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:06.5437664Z dist init r=0, world=2
2022-09-27T16:26:06.5441252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:06.5580534Z dist init r=1, world=2
2022-09-27T16:26:06.5586132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:06.5587241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:06.5645405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:08.0619159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:08.0619727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:08.6326994Z ok (4.012s)
2022-09-27T16:26:08.6350710Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39078
2022-09-27T16:26:08.6357145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39079
2022-09-27T16:26:10.3286089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:10.3286790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:10.3288047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:10.3288542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:10.3376172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:10.3376647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:10.3379619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:10.3380095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:10.5840264Z dist init r=1, world=2
2022-09-27T16:26:10.5844594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:10.5879857Z dist init r=0, world=2
2022-09-27T16:26:10.5885306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:10.5886411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:10.5947501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:12.1282521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:12.1283089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:12.6447993Z ok (4.012s)
2022-09-27T16:26:12.6471397Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39159
2022-09-27T16:26:12.6477884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39160
2022-09-27T16:26:14.3555360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:14.3556196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:14.3557186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:14.3557656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:14.3591842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:14.3592314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:14.3595265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:14.3595735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:14.6078132Z dist init r=0, world=2
2022-09-27T16:26:14.6082672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:14.6083095Z dist init r=1, world=2
2022-09-27T16:26:14.6088646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:14.6089533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:14.6185557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:16.1431796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:16.1432739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:16.6571530Z ok (4.012s)
2022-09-27T16:26:16.6593821Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39240
2022-09-27T16:26:16.6599935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39241
2022-09-27T16:26:18.3561494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:18.3562182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:18.3563214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:18.3563716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:18.3622625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:18.3623105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:18.3626414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:18.3626911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:18.6116180Z dist init r=0, world=2
2022-09-27T16:26:18.6119877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:18.6145099Z dist init r=1, world=2
2022-09-27T16:26:18.6150847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:18.6151768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:18.6222813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:20.1601636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:20.1602621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:20.6690274Z ok (4.012s)
2022-09-27T16:26:20.6713900Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39321
2022-09-27T16:26:20.6720306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39322
2022-09-27T16:26:22.3248904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:22.3250378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:22.3251515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:22.3252488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:22.3402272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:22.3402785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:22.3405261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:22.3405772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:22.5812391Z dist init r=1, world=2
2022-09-27T16:26:22.5816165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:22.5938392Z dist init r=0, world=2
2022-09-27T16:26:22.5944603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:22.5946170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:22.6021075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:24.1295073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:24.1296084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:24.6828609Z ok (4.014s)
2022-09-27T16:26:24.6852890Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39402
2022-09-27T16:26:24.6859257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39403
2022-09-27T16:26:26.3322496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:26.3323112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:26.3323668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:26.3324173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:26.3734687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:26.3735143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:26.3738284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:26.3738775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:26.5839027Z dist init r=1, world=2
2022-09-27T16:26:26.5842912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:26.6171732Z dist init r=0, world=2
2022-09-27T16:26:26.6177344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:26.6178538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:26.6249293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:28.1566633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:28.1567143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:28.6952470Z ok (4.012s)
2022-09-27T16:26:28.6975861Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39483
2022-09-27T16:26:28.6981907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39484
2022-09-27T16:26:30.3908289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:30.3908860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:30.3909451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:30.3909941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:30.4109941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:30.4110394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:30.4113437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:30.4113949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:30.6588697Z dist init r=1, world=2
2022-09-27T16:26:30.6592854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:30.6696270Z dist init r=0, world=2
2022-09-27T16:26:30.6702401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:30.6703569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:30.6797692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:32.1845495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:32.1846046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:32.7070784Z ok (4.012s)
2022-09-27T16:26:32.7099854Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39564
2022-09-27T16:26:32.7106380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39565
2022-09-27T16:26:34.4558582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:34.4559171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:34.4559763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:34.4560222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:34.4575748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:34.4576229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:34.4579051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:34.4579541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:34.7077718Z dist init r=0, world=2
2022-09-27T16:26:34.7082299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:34.7176511Z dist init r=1, world=2
2022-09-27T16:26:34.7181978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:34.7182858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:34.7185031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:36.2844144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:36.2844739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:36.8196344Z ok (4.112s)
2022-09-27T16:26:36.8225337Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39645
2022-09-27T16:26:36.8232101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39646
2022-09-27T16:26:38.4902939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:38.4903449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:38.4904068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:38.4904535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:38.5106968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:38.5107475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:38.5110419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:38.5110890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:38.7451128Z dist init r=0, world=2
2022-09-27T16:26:38.7455845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:38.7566986Z dist init r=1, world=2
2022-09-27T16:26:38.7573570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:38.7574403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:38.7659951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:40.2720756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:40.2721392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:40.2956793Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:26:40.2957530Z   warnings.warn(
2022-09-27T16:26:40.2963791Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2916: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-09-27T16:26:40.2964505Z   warnings.warn(
2022-09-27T16:26:40.8324325Z ok (4.013s)
2022-09-27T16:26:40.8353254Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39726
2022-09-27T16:26:40.8359713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39727
2022-09-27T16:26:42.4977629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:42.4978374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:42.4979085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:42.4979554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:42.5176795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:42.5177458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:42.5180183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:42.5180668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:42.7561636Z dist init r=1, world=2
2022-09-27T16:26:42.7565676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:42.7645996Z dist init r=0, world=2
2022-09-27T16:26:42.7651880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:42.7653647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:42.7668582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:44.3128991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:44.3129811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:44.8450690Z ok (4.012s)
2022-09-27T16:26:44.8478626Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39807
2022-09-27T16:26:44.8485016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39808
2022-09-27T16:26:46.5407652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:46.5408168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:46.5408818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:46.5409318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:46.5692656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:46.5693108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:46.5696947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:46.5697449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:46.8066968Z dist init r=1, world=2
2022-09-27T16:26:46.8071283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:46.8101905Z dist init r=0, world=2
2022-09-27T16:26:46.8107496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:46.8108330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:46.8174585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:48.3360237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:48.3360787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:48.9576867Z ok (4.113s)
2022-09-27T16:26:48.9603628Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39888
2022-09-27T16:26:48.9610033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39889
2022-09-27T16:26:50.6632900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:50.6633453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:50.6634075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:50.6634566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:50.6991286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:50.6991778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:50.6994541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:50.6995038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:50.9191057Z dist init r=0, world=2
2022-09-27T16:26:50.9195934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:50.9439808Z dist init r=1, world=2
2022-09-27T16:26:50.9445372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:50.9446210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:50.9500905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:52.4487901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:52.4488473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:52.4716532Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:26:52.4717464Z   warnings.warn(
2022-09-27T16:26:52.4718594Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:26:52.4719349Z   warnings.warn(
2022-09-27T16:26:53.4713112Z ok (4.513s)
2022-09-27T16:26:53.4740681Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39969
2022-09-27T16:26:53.4747167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39970
2022-09-27T16:26:55.1300643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:55.1301499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:55.1302583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:55.1303122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:55.1819362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:55.1819865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:55.1822501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:55.1823033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:55.3752503Z dist init r=0, world=2
2022-09-27T16:26:55.3756408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:55.4229744Z dist init r=1, world=2
2022-09-27T16:26:55.4235318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:55.4236376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:55.4263925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:56.9388297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:56.9388867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:56.9594123Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:26:56.9594963Z   warnings.warn(
2022-09-27T16:26:56.9596070Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:26:56.9596838Z   warnings.warn(
2022-09-27T16:26:57.9846908Z ok (4.513s)
2022-09-27T16:26:57.9871752Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40050
2022-09-27T16:26:57.9878565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40051
2022-09-27T16:26:59.7419480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:59.7420001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:59.7420608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:59.7421094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:59.7554446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:59.7554906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:59.7557576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:59.7558369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:59.9966198Z dist init r=0, world=2
2022-09-27T16:26:59.9970743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:00.0037593Z dist init r=1, world=2
2022-09-27T16:27:00.0043314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:00.0044517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:00.0073513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:01.5281909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:01.5282510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:01.5474511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:27:01.5475338Z   warnings.warn(
2022-09-27T16:27:01.5476787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:27:01.5477551Z   warnings.warn(
2022-09-27T16:27:02.0972098Z ok (4.112s)
2022-09-27T16:27:02.0994646Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40131
2022-09-27T16:27:03.7463463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:03.7464045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:03.7464666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:03.7465133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:03.9849682Z dist init r=0, world=1
2022-09-27T16:27:03.9854770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:03.9855581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:05.3434684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:05.8070254Z ok (3.710s)
2022-09-27T16:27:05.8091938Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40172
2022-09-27T16:27:07.4909749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.4910272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.4910865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.4911367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:07.7432443Z dist init r=0, world=1
2022-09-27T16:27:07.7436705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:07.7437782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:09.1446239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:09.6169191Z ok (3.810s)
2022-09-27T16:27:09.6191531Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40213
2022-09-27T16:27:11.2673525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:11.2674035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:11.2674906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:11.2675409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:11.5112147Z dist init r=0, world=1
2022-09-27T16:27:11.5116713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:11.5117544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:12.8715791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:13.3270301Z ok (3.710s)
2022-09-27T16:27:13.3291918Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40254
2022-09-27T16:27:15.0073076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:15.0073650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:15.0074287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:15.0074751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:15.2631578Z dist init r=0, world=1
2022-09-27T16:27:15.2637189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:15.2638087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:16.6299977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:17.0365819Z ok (3.709s)
2022-09-27T16:27:17.0387569Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40295
2022-09-27T16:27:18.7253425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:18.7254105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:18.7255087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:18.7255601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:18.9644560Z dist init r=0, world=1
2022-09-27T16:27:18.9649242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:18.9650342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:20.3232571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:20.7462366Z ok (3.710s)
2022-09-27T16:27:20.7483414Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40336
2022-09-27T16:27:22.4193375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:22.4193970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:22.4194767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:22.4195261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:22.6624341Z dist init r=0, world=1
2022-09-27T16:27:22.6628774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:22.6630449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:24.0268114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:24.4557069Z ok (3.709s)
2022-09-27T16:27:24.4579164Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40377
2022-09-27T16:27:26.1269381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:26.1270101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:26.1271269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:26.1272107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:26.3778385Z dist init r=0, world=1
2022-09-27T16:27:26.3783252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:26.3784554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:27.7754937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:28.2656003Z ok (3.810s)
2022-09-27T16:27:28.2676877Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40418
2022-09-27T16:27:29.9334755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:29.9335237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:29.9336804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:29.9337314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:30.1849453Z dist init r=0, world=1
2022-09-27T16:27:30.1854817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:30.1855674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:27:31.5819317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:32.0752996Z ok (3.810s)
2022-09-27T16:27:32.0753250Z 
2022-09-27T16:27:32.0753594Z ----------------------------------------------------------------------
2022-09-27T16:27:32.0756363Z Ran 73 tests in 300.154s
2022-09-27T16:27:32.0756768Z 
2022-09-27T16:27:32.0757074Z OK
2022-09-27T16:27:32.0757237Z 
2022-09-27T16:27:32.0757414Z Generating XML reports...
2022-09-27T16:27:32.0855539Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220927162231.xml
2022-09-27T16:27:32.0865091Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220927162231.xml
2022-09-27T16:27:32.5463144Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-09-27 16:27:32.545693]
2022-09-27T16:27:32.5464015Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:27:32.545793]
2022-09-27T16:27:34.7364332Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-09-27T16:27:34.7393254Z 
2022-09-27T16:27:34.7393506Z Running tests...
2022-09-27T16:27:34.7394067Z ----------------------------------------------------------------------
2022-09-27T16:27:34.7416086Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:36.4005455Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:36.4170270Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-09-27T16:27:36.4185707Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:36.4219122Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40494
2022-09-27T16:27:36.4226306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40495
2022-09-27T16:27:38.3305632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:38.3306761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:38.3307400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:38.3307886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:38.4125392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:38.4126372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:38.4128826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:38.4129911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:38.5323201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:38.5532967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:38.6062920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:38.6273607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:38.6274934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:38.6344173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:42.2359095Z ok (5.819s)
2022-09-27T16:27:42.2370467Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:42.2385560Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40580
2022-09-27T16:27:42.2392024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40581
2022-09-27T16:27:44.2047901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:44.2048468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:44.2049292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:44.2050163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:44.2472613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:44.2473067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:44.2476359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:44.2476851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:44.4086066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:44.4295937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:44.4382214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:44.4599772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:44.4601162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:44.4601951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:46.0045273Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:46.0047698Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:46.4775909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:46.4776449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:46.5203721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:46.5204223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:47.1502704Z ok (4.914s)
2022-09-27T16:27:47.1512564Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:47.1528169Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40695
2022-09-27T16:27:47.1535653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40696
2022-09-27T16:27:49.1114437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:49.1115019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:49.1115612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:49.1116095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:49.1450495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:49.1451230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:49.1454351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:49.1454843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:49.3158484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:49.3365066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:49.3366238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:49.3581896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:49.3582762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:49.3672752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:50.9010069Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:50.9012236Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:51.3573993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:51.3574508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:51.3995814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:51.3996365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:52.0643009Z ok (4.914s)
2022-09-27T16:27:52.0654425Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:52.0669854Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40810
2022-09-27T16:27:52.0676453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40811
2022-09-27T16:27:54.0309816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:54.0310344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:54.0310969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:54.0311391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:54.0311996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:54.0312530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:54.0314523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:54.0314975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:54.2407332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:54.2453575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:54.2624211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:54.2678247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:54.2679105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:54.2727070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:55.8368075Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:55.8371184Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:27:56.2975927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:56.2976480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:56.3429713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:56.3430224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:27:56.9786469Z ok (4.914s)
2022-09-27T16:27:56.9797643Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:27:56.9813367Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40925
2022-09-27T16:27:56.9820274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40926
2022-09-27T16:27:58.9542076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:58.9542646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:58.9543168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:58.9543643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:58.9999897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:59.0000380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:59.0002547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:59.0003014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:59.1542483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:59.1750598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:59.1884101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:59.2095799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:59.2096675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:59.2157381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:00.7579226Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:00.7581632Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:01.2387627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:01.2388631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:01.2863809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:01.2864677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:01.8929163Z ok (4.914s)
2022-09-27T16:28:01.8940779Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:01.8956995Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41040
2022-09-27T16:28:01.8964539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41041
2022-09-27T16:28:03.8860014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:03.8861049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:03.8862276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:03.8863225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:03.9046330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:03.9047312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:03.9049124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:03.9050886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:04.0902145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:04.0964111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:04.1112527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:04.1178296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:04.1179804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:04.1215812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:05.6830507Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:05.6832973Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:06.1310192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:06.1310782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:06.1687682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:06.1688214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:06.8075443Z ok (4.915s)
2022-09-27T16:28:06.8094745Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:06.8111090Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41155
2022-09-27T16:28:06.8337512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41156
2022-09-27T16:28:08.8085384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:08.8085955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:08.8086921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:08.8087412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:08.8324742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:08.8325247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:08.8328810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:08.8329314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:09.0106880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:09.0204238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:09.0316869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:09.0415771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:09.0417217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:09.0419904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:10.6053705Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:10.6056310Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:11.0780978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:11.0781573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:11.1173777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:11.1174332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:11.7447259Z ok (4.937s)
2022-09-27T16:28:11.7458710Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:11.7475018Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41270
2022-09-27T16:28:11.7482602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41271
2022-09-27T16:28:13.7799496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:13.7800135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:13.7801251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:13.7801759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:13.7882326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:13.7883161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:13.7885927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:13.7886419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:13.9775967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:13.9867507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:13.9988929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:14.0085644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:14.0086729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:14.0091450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:15.5426477Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:15.5428951Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:16.0021315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:16.0021833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:16.0452120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:16.0452616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:16.6590436Z ok (4.914s)
2022-09-27T16:28:16.6601107Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:16.6617676Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41385
2022-09-27T16:28:16.6625524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41386
2022-09-27T16:28:18.6465076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:18.6465599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:18.6466252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:18.6466748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:18.6896514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:18.6896983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:18.6899928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:18.6900430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:18.8506780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:18.8716021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:18.8772597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:18.8988522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:18.8989598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:18.9021246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:20.4330708Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:20.4334179Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:20.9059379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:20.9059951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:20.9500323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:20.9500845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:21.5735169Z ok (4.914s)
2022-09-27T16:28:21.5745833Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:21.5761669Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41500
2022-09-27T16:28:21.5769269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41501
2022-09-27T16:28:23.5345828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:23.5346376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:23.5346994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:23.5347488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:23.5570765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:23.5571227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:23.5574482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:23.5575145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:23.7438804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:23.7517868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:23.7648708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:23.7734115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:23.7734941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:23.7750737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:25.3141781Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:25.3144155Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:25.7689502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:25.7690478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:25.8131392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:25.8131898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:26.3895438Z ok (4.816s)
2022-09-27T16:28:26.3906205Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:26.3922721Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41615
2022-09-27T16:28:26.3930611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41616
2022-09-27T16:28:28.3237630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:28.3238147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:28.3238749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:28.3239206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:28.3662569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:28.3663039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:28.3665712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:28.3666191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:28.5262804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:28.5522393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:28.5559473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:28.5771924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:28.5773400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:28.5830286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:30.1052029Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:30.1054355Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:30.5719452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:30.5720039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:30.6174932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:30.6175412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:31.2038272Z ok (4.814s)
2022-09-27T16:28:31.2048762Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:31.2065476Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41730
2022-09-27T16:28:31.2073348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41731
2022-09-27T16:28:33.1421764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:33.1422244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:33.1422888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:33.1423363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:33.2036612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:33.2037086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:33.2039218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:33.2039727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:33.3422146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:33.3631961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:33.3926509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:33.4137836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:33.4138863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:33.4140040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:34.9431218Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:34.9433687Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:35.4019835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:35.4020433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:35.4523920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:35.4524463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:36.1182957Z ok (4.914s)
2022-09-27T16:28:36.1193516Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:36.1210085Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41845
2022-09-27T16:28:36.1218172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41846
2022-09-27T16:28:38.1074262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:38.1074809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:38.1075743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:38.1076246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:38.1225240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:38.1226076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:38.1228438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:38.1228924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:38.3088260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:38.3117458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:38.3301215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:38.3330500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:38.3331320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:38.3403643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:39.8775043Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:39.8777182Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:40.3518198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:40.3519130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:40.4023036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:40.4023570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:41.0327161Z ok (4.914s)
2022-09-27T16:28:41.0337458Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:41.0354130Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41960
2022-09-27T16:28:41.0361567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41961
2022-09-27T16:28:43.0310435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:43.0311026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:43.0311869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:43.0312362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:43.0690152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:43.0690867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:43.0694005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:43.0694490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:43.2388712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:43.2590292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:43.2598145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:43.2800040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:43.2800915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:43.2802717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:44.8418620Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:44.8421650Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:45.3497279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:45.3497825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:45.3915468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:45.3915939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:45.9488761Z ok (4.916s)
2022-09-27T16:28:45.9499998Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:45.9516649Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42075
2022-09-27T16:28:45.9524742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42076
2022-09-27T16:28:47.9732218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:47.9732839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:47.9733999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:47.9734529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:48.0048129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:48.0048588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:48.0051742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:48.0052258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:48.1732028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:48.1930752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:48.1940565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:48.2142986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:48.2144047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:48.2144969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:49.7712965Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:49.7715424Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:50.2235503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:50.2236059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:50.2666142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:50.2666961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:50.8634217Z ok (4.914s)
2022-09-27T16:28:50.8644354Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:50.8661475Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42190
2022-09-27T16:28:50.8669677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42191
2022-09-27T16:28:52.8001929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:52.8003004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:52.8004248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:52.8005193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:52.8498906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:52.8499936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:52.8501552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:52.8502519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:53.0036321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:53.0245506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:53.0396519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:53.0610010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:53.0611619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:53.0651662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:54.6087389Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:54.6090113Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:55.0890518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:55.0891104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:55.1359534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:55.1360515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:55.6777492Z ok (4.814s)
2022-09-27T16:28:55.6788171Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:28:55.6805282Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42305
2022-09-27T16:28:55.6813953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42306
2022-09-27T16:28:57.6403319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:57.6403848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:57.6404954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:57.6405492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:57.6620081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:57.6620543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:57.6623358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:57.6623859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:57.8421070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:57.8499021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:57.8630668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:57.8709092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:57.8710204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:57.8733769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:59.4392218Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:59.4394564Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-09-27T16:28:59.8962582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:59.8963136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:59.9440658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:28:59.9441165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:29:00.4922651Z ok (4.814s)
2022-09-27T16:29:00.4954601Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:00.4971848Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42420
2022-09-27T16:29:00.4981252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42421
2022-09-27T16:29:02.4562243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:02.4562807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:02.4563667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:02.4564140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:02.5161943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:02.5162816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:02.5165068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:02.5165547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:02.6648368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:02.6858334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:02.7076612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:02.7285809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:02.7286652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:02.7366080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:04.1659289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0435iik
2022-09-27T16:29:04.1659936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0435iik/_remote_module_non_scriptable.py
2022-09-27T16:29:04.1662696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83jte9wi
2022-09-27T16:29:04.1665948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83jte9wi/_remote_module_non_scriptable.py
2022-09-27T16:29:04.7420278Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:04.7473232Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:05.0076122Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0091356Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0272370Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0288220Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0470569Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0487020Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0669543Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0684952Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0867401Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.0883511Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1065722Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1081916Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1263946Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1280419Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1575172Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.1606206Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:05.8095786Z ok (5.317s)
2022-09-27T16:29:05.8126168Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:05.8143315Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42505
2022-09-27T16:29:05.8152348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42506
2022-09-27T16:29:07.7938756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:07.7939314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:07.7939997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:07.7940497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:07.8314263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:07.8314786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:07.8317131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:07.8317623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:07.9963687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:08.0171440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:08.0210061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:08.0419821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:08.0420570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:08.0476936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:09.4544196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqzm01bo
2022-09-27T16:29:09.4544863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqzm01bo/_remote_module_non_scriptable.py
2022-09-27T16:29:09.4780487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgz342njv
2022-09-27T16:29:09.4783059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgz342njv/_remote_module_non_scriptable.py
2022-09-27T16:29:10.0179081Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:10.0222651Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:10.2984918Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.2999933Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3185092Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3202696Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3388366Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3406067Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3591422Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3608461Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3793539Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3810983Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.3996228Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.4014370Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.4199709Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.4217293Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.4513381Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:10.4558910Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:11.0268806Z ok (5.217s)
2022-09-27T16:29:11.0299954Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:11.0317776Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42590
2022-09-27T16:29:11.0326154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42591
2022-09-27T16:29:12.9935016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:12.9935864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:12.9937492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:12.9938502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:12.9939108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:12.9939638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:12.9940253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:12.9940677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:13.2008884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:13.2011557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:13.2220901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:13.2223474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:13.2224264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:13.2323330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:14.6391715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9qgjyi0
2022-09-27T16:29:14.6392389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9qgjyi0/_remote_module_non_scriptable.py
2022-09-27T16:29:14.6523170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkenn5j5
2022-09-27T16:29:14.6526110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkenn5j5/_remote_module_non_scriptable.py
2022-09-27T16:29:15.1960234Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:15.2045872Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:15.4698372Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.4699123Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.4883098Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.4891668Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5076874Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5087005Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5272037Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5281653Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5466496Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5477025Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5660817Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5671011Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5856503Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.5865861Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.6158785Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:15.6192315Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:16.2442628Z ok (5.217s)
2022-09-27T16:29:16.2474920Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:16.2492226Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42675
2022-09-27T16:29:16.2501143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42676
2022-09-27T16:29:18.2674395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:18.2674927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:18.2675539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:18.2676033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:18.2862672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:18.2863169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:18.2865968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:18.2866434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:18.4785848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:18.4813929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:18.4996600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:18.5023173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:18.5024468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:18.5099135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:19.9337426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mdbtlcu
2022-09-27T16:29:19.9338258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mdbtlcu/_remote_module_non_scriptable.py
2022-09-27T16:29:19.9442508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx34z32sf
2022-09-27T16:29:19.9445361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx34z32sf/_remote_module_non_scriptable.py
2022-09-27T16:29:20.4831481Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:20.4932460Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:20.7489082Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.7496633Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.7682750Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.7689638Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.7880121Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.7886038Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8077428Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8084234Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8276875Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8287710Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8475086Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8483283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8670322Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8678156Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.8975900Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:20.9015118Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:21.4617010Z ok (5.217s)
2022-09-27T16:29:21.4648453Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:21.4666351Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42760
2022-09-27T16:29:21.4675252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42761
2022-09-27T16:29:23.3913694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:23.3914205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:23.3914765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:23.3915251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:23.4470612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:23.4471049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:23.4473835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:23.4474306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:23.5924683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:23.6133674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:23.6349762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:23.6559557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:23.6560726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:23.6641112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:25.0676895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qumkf3o
2022-09-27T16:29:25.0678075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qumkf3o/_remote_module_non_scriptable.py
2022-09-27T16:29:25.0863666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmfhu3su
2022-09-27T16:29:25.0864781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmfhu3su/_remote_module_non_scriptable.py
2022-09-27T16:29:25.6327606Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:25.6442976Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:25.8945670Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.8967219Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9136529Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9158400Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9327818Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9349697Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9518584Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9541154Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9711351Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9731823Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9901616Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:25.9923522Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:26.0095914Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:26.0115005Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:26.0325906Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:26.0326672Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:26.6790466Z ok (5.217s)
2022-09-27T16:29:26.6822451Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:26.6839766Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42845
2022-09-27T16:29:26.7098784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42846
2022-09-27T16:29:28.6614743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:28.6615302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:28.6615880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:28.6616366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:28.7022601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:28.7023075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:28.7025765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:28.7026259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:28.8607762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:28.8815432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:28.8897010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:28.9107154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:28.9108017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:28.9120346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:30.3324511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb266rbm7
2022-09-27T16:29:30.3325156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb266rbm7/_remote_module_non_scriptable.py
2022-09-27T16:29:30.3457992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphch067lw
2022-09-27T16:29:30.3461235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphch067lw/_remote_module_non_scriptable.py
2022-09-27T16:29:30.8985751Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:30.9087434Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:29:31.1553116Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.1567192Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.1741009Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.1755515Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.1929746Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.1944626Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2118606Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2133150Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2307465Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2322032Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2496672Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2510767Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2685724Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2699793Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2906096Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.2910825Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-09-27T16:29:31.9214724Z ok (5.242s)
2022-09-27T16:29:31.9224265Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:31.9242604Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42930
2022-09-27T16:29:31.9251855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42931
2022-09-27T16:29:33.9282747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:33.9283589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:33.9284858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:33.9285388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:33.9742935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:33.9743801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:33.9746460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:33.9746974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:34.1296570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:34.1505168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:34.1621977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:34.1832988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:34.1833800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:34.1911710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:37.3373439Z ok (5.416s)
2022-09-27T16:29:37.3393803Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:37.3411777Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43016
2022-09-27T16:29:37.3422506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43017
2022-09-27T16:29:39.2725607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:39.2726113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:39.2727168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:39.2727692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:39.2943050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:39.2943885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:39.2946791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:39.2947272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:39.4746349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:39.4838794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:39.4955447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:39.5048237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:39.5049144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:39.5057941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:43.1557060Z ok (5.818s)
2022-09-27T16:29:43.1581267Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:43.1600251Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43102
2022-09-27T16:29:43.1609173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43103
2022-09-27T16:29:45.0992376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:45.0992919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:45.0994302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:45.0994850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:45.1471525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:45.1472029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:45.1474842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:45.1475337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:45.3006265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:45.3212380Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-09-27T16:29:45.3347249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:45.3558428Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-09-27T16:29:45.7669564Z ok (2.611s)
2022-09-27T16:29:45.7681716Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:45.7683077Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:29:45.7696724Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:45.7713811Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43170
2022-09-27T16:29:45.7723107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43171
2022-09-27T16:29:47.7421851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:47.7422719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:47.7423536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:47.7424024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:47.7588541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:47.7589021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:47.7592306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:47.7592803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:47.9497175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:47.9520890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:47.9709019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:47.9731989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:47.9733889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:47.9811390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:50.4828508Z ok (4.715s)
2022-09-27T16:29:50.4844829Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:50.4864978Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43255
2022-09-27T16:29:50.4873515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43256
2022-09-27T16:29:52.4313873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:52.4314404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:52.4315015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:52.4315478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:52.4755197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:52.4755664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:52.4758710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:52.4759194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:52.6327720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:52.6536447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:52.6636477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:52.6846978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:52.6847793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:52.6943650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:55.1979224Z ok (4.715s)
2022-09-27T16:29:55.1982172Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:55.2000549Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43340
2022-09-27T16:29:55.2009928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43341
2022-09-27T16:29:57.1436685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:57.1437230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:57.1437858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:57.1440430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:57.1875827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:57.1876326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:57.1879389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:57.1879909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:57.3493946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:57.3773692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:57.4117186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:57.4118335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:57.4119540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:57.4120262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:57.4235510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7zczpcr
2022-09-27T16:29:57.4236081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0w9o6n_
2022-09-27T16:29:57.4238124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7zczpcr/_remote_module_non_scriptable.py
2022-09-27T16:29:57.4238698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0w9o6n_/_remote_module_non_scriptable.py
2022-09-27T16:29:57.4478344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:29:57.4478850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:29:57.4897628Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpj0w9o6n_'>
2022-09-27T16:29:57.4898120Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:29:57.4898638Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpd7zczpcr'>
2022-09-27T16:29:57.4899140Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:29:57.9070766Z ok (2.709s)
2022-09-27T16:29:57.9074398Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:29:57.9094209Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43418
2022-09-27T16:29:57.9103599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43419
2022-09-27T16:29:59.8874215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:59.8874732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:59.8876080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:59.8877139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:59.9073869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:59.9074522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:59.9077108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:59.9077851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:00.0972495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:00.1034783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:00.1188943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:00.1250994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:00.1252269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:00.1291888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:01.5477251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh70g22t
2022-09-27T16:30:01.5478219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh70g22t/_remote_module_non_scriptable.py
2022-09-27T16:30:01.5731918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1ly554n
2022-09-27T16:30:01.5733258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1ly554n/_remote_module_non_scriptable.py
2022-09-27T16:30:03.2093517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:30:03.2094116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:30:03.2856111Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpsh70g22t'>
2022-09-27T16:30:03.2856665Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:30:03.2857282Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpm1ly554n'>
2022-09-27T16:30:03.2857752Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:30:03.9238437Z ok (6.017s)
2022-09-27T16:30:03.9244464Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:30:03.9264080Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43504
2022-09-27T16:30:03.9273447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43505
2022-09-27T16:30:05.8985609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:05.8986164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:05.8986774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:05.8987270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:05.9055528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:05.9056000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:05.9058779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:05.9059256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:06.0883641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:06.1019158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:06.5335986Z skip: Need at least 4 CUDA devices (2.610s)
2022-09-27T16:30:06.5341821Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-09-27T16:30:06.5361836Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43572
2022-09-27T16:30:06.5371607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43573
2022-09-27T16:30:08.4642231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:08.4642817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:08.4646432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:08.4646876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:08.4869215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:08.4869971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:08.4872133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:08.4872598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:08.6647475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:08.6748030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:09.1432571Z skip: Need at least 4 CUDA devices (2.610s)
2022-09-27T16:30:09.1448068Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:09.1466893Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43640
2022-09-27T16:30:11.0473728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:11.0474297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:11.0475531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:11.0476077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:11.2513256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:11.2518095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:11.2519000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:11.5515927Z ok (2.408s)
2022-09-27T16:30:11.5525266Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:11.5544557Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43676
2022-09-27T16:30:13.5025484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:13.5026024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:13.5026930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:13.5027422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:13.7073048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:13.7078301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:13.7079341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:15.4629441Z ok (3.911s)
2022-09-27T16:30:15.4636465Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:15.4656046Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43719
2022-09-27T16:30:17.3922364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:17.3922983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:17.3923929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:17.3924427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:17.5953276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:17.5957427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:17.5958524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:17.8706190Z ok (2.408s)
2022-09-27T16:30:17.8727460Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:17.8747759Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43755
2022-09-27T16:30:19.8677109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:19.8677670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:19.8678261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:19.8678739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:20.0750908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:20.0755725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:20.0756487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:21.8828399Z ok (4.012s)
2022-09-27T16:30:21.8836587Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:21.8856131Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43798
2022-09-27T16:30:23.8539783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:23.8540338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:23.8540950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:23.8541436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:24.0626428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:24.0631129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:24.0632334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:25.8938420Z ok (4.011s)
2022-09-27T16:30:25.8946221Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:25.8965725Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43841
2022-09-27T16:30:27.8427870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:27.8428441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:27.8429329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:27.8429830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:28.0518666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:28.0523727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:28.0524516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:29.9049948Z ok (4.011s)
2022-09-27T16:30:29.9057936Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:29.9078154Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43884
2022-09-27T16:30:31.8834344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:31.8835471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:31.8836098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:31.8836579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:32.0937845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:32.0942328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:32.0943808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:33.9163731Z ok (4.011s)
2022-09-27T16:30:33.9171472Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-09-27T16:30:33.9191876Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43927
2022-09-27T16:30:35.9028866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:35.9029890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:35.9030526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:35.9031039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:36.1130712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:36.1135617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:36.1137076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:30:36.4243650Z ok (2.508s)
2022-09-27T16:30:36.4243930Z 
2022-09-27T16:30:36.4246462Z ----------------------------------------------------------------------
2022-09-27T16:30:36.4247187Z Ran 42 tests in 181.685s
2022-09-27T16:30:36.4247539Z 
2022-09-27T16:30:36.4247759Z OK (skipped=4)
2022-09-27T16:30:36.4248083Z 
2022-09-27T16:30:36.4248306Z Generating XML reports...
2022-09-27T16:30:36.4327849Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220927162734.xml
2022-09-27T16:30:36.4351012Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220927162734.xml
2022-09-27T16:30:36.8653826Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-09-27 16:30:36.864819]
2022-09-27T16:30:36.8654617Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:30:36.864928]
2022-09-27T16:30:38.7407356Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2022-09-27T16:30:38.7427189Z 
2022-09-27T16:30:38.7427337Z Running tests...
2022-09-27T16:30:38.7428055Z ----------------------------------------------------------------------
2022-09-27T16:30:40.3704786Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:40.3894094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44000
2022-09-27T16:30:40.3900720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44001
2022-09-27T16:30:42.0598759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:42.0599615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:42.0600410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:42.0600863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:42.0868225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:42.0868687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:42.0872611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:42.0873053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:42.3396730Z dist init r=1, world=2
2022-09-27T16:30:42.3401343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:42.3513007Z dist init r=0, world=2
2022-09-27T16:30:42.3519245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:42.3520051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:42.3606459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:43.8716245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:43.8716823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:44.9000521Z ok (6.157s)
2022-09-27T16:30:44.9024231Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44085
2022-09-27T16:30:44.9030335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44086
2022-09-27T16:30:46.6256192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:46.6257088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:46.6258400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:46.6258903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:46.6497561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:46.6498003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:46.6502359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:46.6502834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:46.9124451Z dist init r=0, world=2
2022-09-27T16:30:46.9130075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:46.9134002Z dist init r=1, world=2
2022-09-27T16:30:46.9140630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:46.9141475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:46.9233052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:48.4091396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:48.4092041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:49.5132698Z ok (4.613s)
2022-09-27T16:30:49.5156455Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44170
2022-09-27T16:30:49.5165230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44171
2022-09-27T16:30:51.1924724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:51.1925324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:51.1927232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:51.1927707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:51.2248598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:51.2249061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:51.2253195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:51.2253670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:51.4673962Z dist init r=0, world=2
2022-09-27T16:30:51.4678648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:51.4881803Z dist init r=1, world=2
2022-09-27T16:30:51.4888137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:51.4889137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:51.4984473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:52.9794905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:52.9795420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:54.0263121Z ok (4.513s)
2022-09-27T16:30:54.0286363Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44255
2022-09-27T16:30:54.0292970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44256
2022-09-27T16:30:55.7210035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:55.7211127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:55.7212253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:55.7213546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:55.7450210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:55.7450716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:55.7454366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:55.7454806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:56.0011959Z dist init r=0, world=2
2022-09-27T16:30:56.0016432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:56.0110793Z dist init r=1, world=2
2022-09-27T16:30:56.0116874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:56.0117915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:56.0119017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:57.5076290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:57.5077126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:58.5393611Z ok (4.513s)
2022-09-27T16:30:58.5416190Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44340
2022-09-27T16:30:58.5422868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44341
2022-09-27T16:31:00.2191038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:00.2192097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:00.2193319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:00.2194296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:00.2590490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:00.2591425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:00.2596029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:00.2597038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:00.4939887Z dist init r=0, world=2
2022-09-27T16:31:00.4944168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:00.5246795Z dist init r=1, world=2
2022-09-27T16:31:00.5253804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:00.5255258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:00.5351472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:02.0174169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:02.0175159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:03.0525152Z ok (4.513s)
2022-09-27T16:31:03.0547375Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44425
2022-09-27T16:31:03.0554027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44426
2022-09-27T16:31:04.7416912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:04.7417940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:04.7419185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:04.7420158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:04.7781557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:04.7782536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:04.7787178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:04.7788207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:05.0130475Z dist init r=1, world=2
2022-09-27T16:31:05.0135137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:05.0405693Z dist init r=0, world=2
2022-09-27T16:31:05.0412233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:05.0414121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:05.0440823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:06.5611658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:06.5613172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:07.6654804Z ok (4.613s)
2022-09-27T16:31:07.6667466Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:07.6683130Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44510
2022-09-27T16:31:07.6689658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44511
2022-09-27T16:31:09.3378859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:09.3379438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:09.3380836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:09.3381338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:09.3679342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:09.3679816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:09.3684082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:09.3684582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:09.6152467Z dist init r=0, world=2
2022-09-27T16:31:09.6157336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:09.6340475Z dist init r=1, world=2
2022-09-27T16:31:09.6346870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:09.6347880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:09.6361633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:11.1520286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:11.1520857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:12.1787215Z ok (4.513s)
2022-09-27T16:31:12.1800278Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:12.1816787Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44595
2022-09-27T16:31:12.1823299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44596
2022-09-27T16:31:13.8974390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:13.8974928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:13.8976160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:13.8976658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:13.9008630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:13.9009066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:13.9013009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:13.9013503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:14.1581893Z dist init r=1, world=2
2022-09-27T16:31:14.1586297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:14.1811962Z dist init r=0, world=2
2022-09-27T16:31:14.1818359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:14.1819264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:14.1892371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:15.7055158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:15.7055722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:16.7926703Z ok (4.614s)
2022-09-27T16:31:16.7941509Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:31:16.7957517Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44680
2022-09-27T16:31:16.7963882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44681
2022-09-27T16:31:18.4859336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:18.4860026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:18.4861036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:18.4861763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:18.5178207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:18.5178635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:18.5182831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:18.5183338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:18.7571295Z dist init r=1, world=2
2022-09-27T16:31:18.7576362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:18.7823807Z dist init r=0, world=2
2022-09-27T16:31:18.7830067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:18.7830909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:18.7882592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:20.2628612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:20.2629172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:21.3062990Z ok (4.513s)
2022-09-27T16:31:21.3081046Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:21.3097440Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44765
2022-09-27T16:31:21.3103897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44766
2022-09-27T16:31:23.0180990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:23.0181689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:23.0182735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:23.0183511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:23.0736099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:23.0736625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:23.0739846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:23.0740327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:23.2861948Z dist init r=1, world=2
2022-09-27T16:31:23.2866711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:23.3319736Z dist init r=0, world=2
2022-09-27T16:31:23.3326041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:23.3326936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:23.3375242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:24.8375341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:24.8375870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:25.2968100Z Communication hook should not be None
2022-09-27T16:31:25.2978905Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:25.2979365Z     return func(*args, **kwargs)
2022-09-27T16:31:25.2979914Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:25.2980332Z     p_assert(
2022-09-27T16:31:25.2982881Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:25.2983315Z     traceback.print_stack()
2022-09-27T16:31:25.3073988Z Communication hook should not be None
2022-09-27T16:31:25.3086187Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:25.3087135Z     return func(*args, **kwargs)
2022-09-27T16:31:25.3087734Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:25.3088133Z     p_assert(
2022-09-27T16:31:25.3088614Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:25.3089016Z     traceback.print_stack()
2022-09-27T16:31:25.3117089Z Communication hook state should not be None
2022-09-27T16:31:25.3117864Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:25.3118496Z     return func(*args, **kwargs)
2022-09-27T16:31:25.3119097Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:25.3119774Z     p_assert(
2022-09-27T16:31:25.3120280Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:25.3120689Z     traceback.print_stack()
2022-09-27T16:31:25.3121001Z Communication hook state should not be None
2022-09-27T16:31:25.3121509Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:25.3121904Z     return func(*args, **kwargs)
2022-09-27T16:31:25.3123186Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:25.3123626Z     p_assert(
2022-09-27T16:31:25.3124124Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:25.3124524Z     traceback.print_stack()
2022-09-27T16:31:25.8203008Z ok (4.514s)
2022-09-27T16:31:25.8220601Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:25.8236450Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44850
2022-09-27T16:31:25.8242672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44851
2022-09-27T16:31:27.5107819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:27.5108358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:27.5109789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:27.5110320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:27.5452673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:27.5453132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:27.5457466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:27.5457988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:27.7858811Z dist init r=0, world=2
2022-09-27T16:31:27.7862986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:27.8087454Z dist init r=1, world=2
2022-09-27T16:31:27.8093910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:27.8094809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:27.8168443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:29.3125535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:29.3126489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:29.7737177Z Communication hook should not be None
2022-09-27T16:31:29.7747964Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:29.7748381Z     return func(*args, **kwargs)
2022-09-27T16:31:29.7748960Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:29.7749377Z     p_assert(
2022-09-27T16:31:29.7749872Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:29.7750249Z     traceback.print_stack()
2022-09-27T16:31:29.7816918Z Communication hook should not be None
2022-09-27T16:31:29.7828818Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:29.7829314Z     return func(*args, **kwargs)
2022-09-27T16:31:29.7829892Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:29.7830293Z     p_assert(
2022-09-27T16:31:29.7830780Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:29.7831155Z     traceback.print_stack()
2022-09-27T16:31:29.7856253Z Communication hook state should not be None
2022-09-27T16:31:29.7856896Z Communication hook state should not be None
2022-09-27T16:31:29.7857590Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:29.7857985Z     return func(*args, **kwargs)
2022-09-27T16:31:29.7859084Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:29.7859754Z     p_assert(
2022-09-27T16:31:29.7860650Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:29.7862374Z     traceback.print_stack()
2022-09-27T16:31:29.7863010Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:29.7863432Z     return func(*args, **kwargs)
2022-09-27T16:31:29.7863999Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:29.7864425Z     p_assert(
2022-09-27T16:31:29.7864921Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:29.7865328Z     traceback.print_stack()
2022-09-27T16:31:30.3343309Z ok (4.514s)
2022-09-27T16:31:30.3361494Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:31:30.3377551Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44935
2022-09-27T16:31:30.3383897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44936
2022-09-27T16:31:31.9656528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:31.9657583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:31.9658707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:31.9659688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:32.0444272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:32.0444829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:32.0446155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:32.0446897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:32.2379075Z dist init r=0, world=2
2022-09-27T16:31:32.2383458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:32.3024895Z dist init r=1, world=2
2022-09-27T16:31:32.3055904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:32.3057218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:32.3094711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:33.7947220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:33.7947784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:34.2541094Z Communication hook should not be None
2022-09-27T16:31:34.2551955Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:34.2552365Z     return func(*args, **kwargs)
2022-09-27T16:31:34.2552894Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:34.2553307Z     p_assert(
2022-09-27T16:31:34.2553799Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:34.2554176Z     traceback.print_stack()
2022-09-27T16:31:34.2584624Z Communication hook should not be None
2022-09-27T16:31:34.2596748Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:34.2597134Z     return func(*args, **kwargs)
2022-09-27T16:31:34.2597677Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:34.2598087Z     p_assert(
2022-09-27T16:31:34.2598571Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:34.2598944Z     traceback.print_stack()
2022-09-27T16:31:34.2627318Z Communication hook state should not be None
2022-09-27T16:31:34.2627706Z Communication hook state should not be None
2022-09-27T16:31:34.2628225Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:34.2628597Z     return func(*args, **kwargs)
2022-09-27T16:31:34.2629155Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:34.2629578Z     p_assert(
2022-09-27T16:31:34.2630028Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:34.2630419Z     traceback.print_stack()
2022-09-27T16:31:34.2630901Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:34.2631271Z     return func(*args, **kwargs)
2022-09-27T16:31:34.2631839Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:34.2632246Z     p_assert(
2022-09-27T16:31:34.2632723Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:34.2633093Z     traceback.print_stack()
2022-09-27T16:31:34.8483613Z ok (4.514s)
2022-09-27T16:31:34.8501786Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:34.8517227Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45020
2022-09-27T16:31:34.8523570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45021
2022-09-27T16:31:36.5447232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:36.5448255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:36.5449410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:36.5450689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:36.5886431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:36.5887459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:36.5890226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:36.5891259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:36.8283319Z dist init r=0, world=2
2022-09-27T16:31:36.8288061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:36.8505630Z dist init r=1, world=2
2022-09-27T16:31:36.8511289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:36.8512433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:36.8593423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:38.3326011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:38.3327031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:38.8019014Z Communication hook should not be None
2022-09-27T16:31:38.8031429Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:38.8031907Z     return func(*args, **kwargs)
2022-09-27T16:31:38.8032464Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:38.8032849Z     p_assert(
2022-09-27T16:31:38.8033335Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:38.8033740Z     traceback.print_stack()
2022-09-27T16:31:38.8169347Z Communication hook should not be None
2022-09-27T16:31:38.8184100Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:38.8184824Z     return func(*args, **kwargs)
2022-09-27T16:31:38.8185917Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:38.8186689Z     p_assert(
2022-09-27T16:31:38.8187591Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:38.8188278Z     traceback.print_stack()
2022-09-27T16:31:38.8245107Z Communication hook state should not be None
2022-09-27T16:31:38.8246174Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:38.8246819Z     return func(*args, **kwargs)
2022-09-27T16:31:38.8247291Z Communication hook state should not be None
2022-09-27T16:31:38.8248275Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:38.8249007Z     p_assert(
2022-09-27T16:31:38.8250273Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:38.8251370Z     traceback.print_stack()
2022-09-27T16:31:38.8252301Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:38.8252999Z     return func(*args, **kwargs)
2022-09-27T16:31:38.8254065Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:38.8254757Z     p_assert(
2022-09-27T16:31:38.8255781Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:38.8256581Z     traceback.print_stack()
2022-09-27T16:31:39.3623195Z ok (4.514s)
2022-09-27T16:31:39.3639202Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:31:39.3655288Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45105
2022-09-27T16:31:39.3661315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45106
2022-09-27T16:31:41.0936364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:41.0936879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:41.0938517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:41.0938930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:41.0993090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:41.0993489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:41.0997458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:41.0997979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:41.3614461Z dist init r=1, world=2
2022-09-27T16:31:41.3618940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:41.3670124Z dist init r=0, world=2
2022-09-27T16:31:41.3676780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:41.3678168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:41.3722925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:42.8542368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:42.8542929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:43.3426496Z Communication hook should not be None
2022-09-27T16:31:43.3426898Z Communication hook should not be None
2022-09-27T16:31:43.3436963Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:43.3437698Z     return func(*args, **kwargs)
2022-09-27T16:31:43.3438358Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:43.3438696Z     return func(*args, **kwargs)
2022-09-27T16:31:43.3439228Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:43.3439618Z     p_assert(
2022-09-27T16:31:43.3440164Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:43.3440566Z     p_assert(
2022-09-27T16:31:43.3441055Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:43.3441756Z     traceback.print_stack()
2022-09-27T16:31:43.3442249Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:43.3442626Z     traceback.print_stack()
2022-09-27T16:31:43.3484462Z Communication hook state should not be None
2022-09-27T16:31:43.3485093Z Communication hook state should not be None
2022-09-27T16:31:43.3485799Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:43.3486150Z     return func(*args, **kwargs)
2022-09-27T16:31:43.3486719Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:43.3487120Z     p_assert(
2022-09-27T16:31:43.3487587Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:43.3488004Z     traceback.print_stack()
2022-09-27T16:31:43.3488487Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:43.3488858Z     return func(*args, **kwargs)
2022-09-27T16:31:43.3489422Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:43.3490307Z     p_assert(
2022-09-27T16:31:43.3490801Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:43.3491170Z     traceback.print_stack()
2022-09-27T16:31:43.9764943Z ok (4.614s)
2022-09-27T16:31:43.9782682Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:31:43.9798998Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45190
2022-09-27T16:31:43.9805195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45191
2022-09-27T16:31:45.6838167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:45.6838717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:45.6840112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:45.6840608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:45.7016450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:45.7016926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:45.7020455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:45.7020960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:45.9670441Z dist init r=1, world=2
2022-09-27T16:31:45.9675363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:45.9740289Z dist init r=0, world=2
2022-09-27T16:31:45.9746449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:45.9747921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:45.9778315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:47.4961666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:47.4962258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:47.9712651Z Communication hook should not be None
2022-09-27T16:31:47.9713072Z Communication hook should not be None
2022-09-27T16:31:47.9724142Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:47.9724703Z     return func(*args, **kwargs)
2022-09-27T16:31:47.9725343Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:47.9725741Z     return func(*args, **kwargs)
2022-09-27T16:31:47.9726302Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:47.9726711Z     p_assert(
2022-09-27T16:31:47.9727260Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3236, in _post_backward_hook
2022-09-27T16:31:47.9727642Z     p_assert(
2022-09-27T16:31:47.9728143Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:47.9728535Z     traceback.print_stack()
2022-09-27T16:31:47.9729044Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:47.9729412Z     traceback.print_stack()
2022-09-27T16:31:47.9776535Z Communication hook state should not be None
2022-09-27T16:31:47.9777285Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:47.9777715Z     return func(*args, **kwargs)
2022-09-27T16:31:47.9778294Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:47.9778715Z     p_assert(
2022-09-27T16:31:47.9779437Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:47.9779956Z     traceback.print_stack()
2022-09-27T16:31:47.9780276Z Communication hook state should not be None
2022-09-27T16:31:47.9780831Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-09-27T16:31:47.9781258Z     return func(*args, **kwargs)
2022-09-27T16:31:47.9781830Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3240, in _post_backward_hook
2022-09-27T16:31:47.9782230Z     p_assert(
2022-09-27T16:31:47.9782713Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-09-27T16:31:47.9783083Z     traceback.print_stack()
2022-09-27T16:31:48.5910677Z ok (4.614s)
2022-09-27T16:31:48.5934095Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45275
2022-09-27T16:31:48.5940310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45276
2022-09-27T16:31:50.2737203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:50.2737706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:50.2739154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:50.2739633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:50.3002292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:50.3002716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:50.3006296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:50.3006795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:50.5512727Z dist init r=0, world=2
2022-09-27T16:31:50.5517599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:50.5571250Z dist init r=1, world=2
2022-09-27T16:31:50.5578662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:50.5579479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:50.5620679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:52.0421787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:52.0422333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:53.1041558Z ok (4.513s)
2022-09-27T16:31:53.1062627Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45360
2022-09-27T16:31:53.1068745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45361
2022-09-27T16:31:54.8115874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:54.8116332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:54.8117001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:54.8117444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:54.8230649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:54.8231053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:54.8236448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:54.8236909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:55.0809246Z dist init r=1, world=2
2022-09-27T16:31:55.0814161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:55.0897691Z dist init r=0, world=2
2022-09-27T16:31:55.0903821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:55.0904653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:55.0916774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:56.5743714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:56.5744306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:57.6168329Z ok (4.513s)
2022-09-27T16:31:57.6190413Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45445
2022-09-27T16:31:57.6196930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45446
2022-09-27T16:31:59.3236400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:59.3237199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:59.3239166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:59.3239666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:59.3413142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:59.3413596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:59.3417593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:59.3418086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:59.6057151Z dist init r=0, world=2
2022-09-27T16:31:59.6060504Z dist init r=1, world=2
2022-09-27T16:31:59.6061224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:59.6066585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:59.6067775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:59.6165134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:01.0876964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:01.0877563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:02.2310197Z ok (4.614s)
2022-09-27T16:32:02.2333412Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45530
2022-09-27T16:32:02.2338969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45531
2022-09-27T16:32:03.9502560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:03.9503111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:03.9504359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:03.9504861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:03.9659015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:03.9659520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:03.9663358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:03.9663850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:04.2292020Z dist init r=1, world=2
2022-09-27T16:32:04.2297053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:04.2393678Z dist init r=0, world=2
2022-09-27T16:32:04.2399348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:04.2400410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:04.2401089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:05.7690404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:05.7690983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:06.8448241Z ok (4.614s)
2022-09-27T16:32:06.8471202Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45615
2022-09-27T16:32:06.8477556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45616
2022-09-27T16:32:08.5567529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:08.5568068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:08.5571452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:08.5572190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:08.5829624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:08.5830125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:08.5833211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:08.5833697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:08.8495965Z dist init r=0, world=2
2022-09-27T16:32:08.8496317Z dist init r=1, world=2
2022-09-27T16:32:08.8501188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:08.8501740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:08.8502592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:08.8503257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:10.3723578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:10.3724560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:11.5588475Z ok (4.714s)
2022-09-27T16:32:11.5610849Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45700
2022-09-27T16:32:11.5617662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45701
2022-09-27T16:32:13.3118948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:13.3119633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:13.3121916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:13.3122642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:13.3699758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:13.3700440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:13.3702000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:13.3702536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:13.6021202Z dist init r=0, world=2
2022-09-27T16:32:13.6026387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:13.6350795Z dist init r=1, world=2
2022-09-27T16:32:13.6356702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:13.6357562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:13.6433528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:15.1719391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:15.1719954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:16.2725380Z ok (4.714s)
2022-09-27T16:32:16.2733932Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:32:16.2750331Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45785
2022-09-27T16:32:16.2756682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45786
2022-09-27T16:32:18.0174267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:18.0175299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:18.0176497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:18.0177402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:18.0574913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:18.0575839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:18.0578450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:18.0579441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:18.3002915Z dist init r=1, world=2
2022-09-27T16:32:18.3008018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:18.3155055Z dist init r=0, world=2
2022-09-27T16:32:18.3160856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:18.3161719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:18.3213666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:19.8338686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:19.8339708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:20.3854330Z ok (4.113s)
2022-09-27T16:32:20.3861852Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:32:20.3877554Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45866
2022-09-27T16:32:20.3883728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45867
2022-09-27T16:32:22.0837552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:22.0838078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:22.0839398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:22.0840268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:22.1118165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:22.1118632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:22.1122310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:22.1122813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:22.3658855Z dist init r=0, world=2
2022-09-27T16:32:22.3663552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:22.3667682Z dist init r=1, world=2
2022-09-27T16:32:22.3673662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:22.3674781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:22.3766511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:23.8557438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:23.8558002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:24.4980725Z ok (4.113s)
2022-09-27T16:32:24.4989005Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:32:24.5005076Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45947
2022-09-27T16:32:24.5011322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45948
2022-09-27T16:32:26.1559621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:26.1560219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:26.1561650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:26.1562182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:26.1891639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:26.1892109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:26.1897432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:26.1897932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:26.4298083Z dist init r=1, world=2
2022-09-27T16:32:26.4302570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:26.4478979Z dist init r=0, world=2
2022-09-27T16:32:26.4485051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:26.4486707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:26.4506406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:27.9527849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:27.9528417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:28.5099825Z ok (4.012s)
2022-09-27T16:32:28.5111737Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:32:28.5127463Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46028
2022-09-27T16:32:28.5134148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46029
2022-09-27T16:32:30.1813790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:30.1814318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:30.1816145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:30.1816621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:30.2030356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:30.2030825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:30.2034459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:30.2035206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:30.4588752Z dist init r=1, world=2
2022-09-27T16:32:30.4593396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:30.4617464Z dist init r=0, world=2
2022-09-27T16:32:30.4623618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:30.4624532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:30.4697749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:31.9517270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:31.9517817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:32.5220277Z ok (4.012s)
2022-09-27T16:32:32.5231069Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-09-27T16:32:32.5246845Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46109
2022-09-27T16:32:32.5253680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46110
2022-09-27T16:32:34.2136444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:34.2137031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:34.2138767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:34.2139514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:34.2437865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:34.2438375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:34.2442630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:34.2443122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:34.4895336Z dist init r=1, world=2
2022-09-27T16:32:34.4899861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:34.5001813Z dist init r=0, world=2
2022-09-27T16:32:34.5008013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:34.5009457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:34.5104698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:36.0008999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:36.0009876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:36.6345459Z ok (4.112s)
2022-09-27T16:32:36.6355745Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-09-27T16:32:36.6371593Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46190
2022-09-27T16:32:36.6378245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46191
2022-09-27T16:32:38.2986673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:38.2987263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:38.2988409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:38.2989185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:38.3334043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:38.3334539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:38.3338341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:38.3338840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:38.5778946Z dist init r=1, world=2
2022-09-27T16:32:38.5783450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:38.5912337Z dist init r=0, world=2
2022-09-27T16:32:38.5918160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:38.5919295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:38.5987997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:40.1008889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:40.1009460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:40.6467380Z ok (4.012s)
2022-09-27T16:32:40.6467691Z 
2022-09-27T16:32:40.6471442Z ----------------------------------------------------------------------
2022-09-27T16:32:40.6471789Z Ran 27 tests in 121.904s
2022-09-27T16:32:40.6472007Z 
2022-09-27T16:32:40.6472101Z OK
2022-09-27T16:32:40.6474288Z 
2022-09-27T16:32:40.6474729Z Generating XML reports...
2022-09-27T16:32:40.6554002Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220927163038.xml
2022-09-27T16:32:41.1195733Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-09-27 16:32:41.119001]
2022-09-27T16:32:41.1196542Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:32:41.119101]
2022-09-27T16:32:43.0767191Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-09-27T16:32:43.0786177Z 
2022-09-27T16:32:43.0786498Z Running tests...
2022-09-27T16:32:43.0786962Z ----------------------------------------------------------------------
2022-09-27T16:32:43.0793197Z   test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-09-27T16:32:44.7235392Z Test grad norm cal API. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:44.7425317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46306
2022-09-27T16:32:44.7431723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46307
2022-09-27T16:32:46.4115212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:46.4115742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:46.4116943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:46.4117468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:46.4409127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:46.4409748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:46.4412970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:46.4413717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:46.6915303Z dist init r=1, world=2
2022-09-27T16:32:46.6920063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:46.6978516Z dist init r=0, world=2
2022-09-27T16:32:46.6984739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:46.6985590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:46.7022668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:48.1926877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:48.1927461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:49.2536727Z ok (6.175s)
2022-09-27T16:32:49.2543411Z   test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-09-27T16:32:49.2559339Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46391
2022-09-27T16:32:49.2566021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46392
2022-09-27T16:32:50.9616655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:50.9617117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:50.9617725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:50.9618213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:50.9856107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:50.9856633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:50.9859022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:50.9859518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:51.2378290Z dist init r=1, world=2
2022-09-27T16:32:51.2383441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:51.2434916Z dist init r=0, world=2
2022-09-27T16:32:51.2440883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:51.2441700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:51.2486078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:52.7118291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:52.7118902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:53.7681733Z ok (4.514s)
2022-09-27T16:32:53.7687897Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-09-27T16:32:53.7704279Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46476
2022-09-27T16:32:53.7710523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46477
2022-09-27T16:32:55.4766404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:55.4767159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:55.4767753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:55.4768788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:55.5468198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:55.5468746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:55.5469733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:55.5470449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:55.7448361Z dist init r=0, world=2
2022-09-27T16:32:55.7453091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:55.8013068Z dist init r=1, world=2
2022-09-27T16:32:55.8019694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:55.8020591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:55.8061772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:57.3215648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:57.3216188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:58.3813379Z ok (4.613s)
2022-09-27T16:32:58.3824567Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-09-27T16:32:58.3840009Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46561
2022-09-27T16:32:58.3846525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46562
2022-09-27T16:32:59.9986368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:59.9986942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:59.9988081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:59.9988599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:00.0854552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:00.0855187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:00.0855838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:00.0856441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:00.2701675Z dist init r=1, world=2
2022-09-27T16:33:00.2706059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:00.3331202Z dist init r=0, world=2
2022-09-27T16:33:00.3337358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:00.3338302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:00.3417652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:01.8262478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:01.8263018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:02.8946518Z ok (4.513s)
2022-09-27T16:33:02.8952296Z   test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-09-27T16:33:02.8968088Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46646
2022-09-27T16:33:02.8975134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46647
2022-09-27T16:33:04.6009010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:04.6009574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:04.6010481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:04.6011012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:04.6423453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:04.6424036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:04.6425878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:04.6426371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:04.8796296Z dist init r=0, world=2
2022-09-27T16:33:04.8800483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:04.8988421Z dist init r=1, world=2
2022-09-27T16:33:04.8995052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:04.8995831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:04.9004370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:06.3773350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:06.3774258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:07.4080050Z ok (4.513s)
2022-09-27T16:33:07.4085872Z   test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-09-27T16:33:07.4102332Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46731
2022-09-27T16:33:07.4108833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46732
2022-09-27T16:33:09.1276839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:09.1277920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:09.1279127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:09.1279968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:09.1498230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:09.1499275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:09.1501504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:09.1502382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:09.4074216Z dist init r=0, world=2
2022-09-27T16:33:09.4079004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:09.4111976Z dist init r=1, world=2
2022-09-27T16:33:09.4118331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:09.4119210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:09.4182185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:10.8910516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:10.8911960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:11.9223301Z ok (4.514s)
2022-09-27T16:33:11.9229094Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-09-27T16:33:11.9245071Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46816
2022-09-27T16:33:11.9251845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46817
2022-09-27T16:33:13.6176064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:13.6176624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:13.6177224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:13.6177721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:13.7038167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:13.7039194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:13.7040353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:13.7041314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:13.8827469Z dist init r=0, world=2
2022-09-27T16:33:13.8832268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:13.9525437Z dist init r=1, world=2
2022-09-27T16:33:13.9531217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:13.9532686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:13.9542505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:15.4375394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:15.8970517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:15.8972699Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:15.8974186Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:15.8976026Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:15.8977387Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:16.4362658Z ok (4.514s)
2022-09-27T16:33:16.4368385Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-09-27T16:33:16.4384534Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46901
2022-09-27T16:33:16.4390916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46902
2022-09-27T16:33:18.1274282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:18.1274875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:18.1275488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:18.1276215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:18.1551713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:18.1552187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:18.1555224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:18.1555714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:18.4060828Z dist init r=0, world=2
2022-09-27T16:33:18.4066125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:18.4116991Z dist init r=1, world=2
2022-09-27T16:33:18.4122725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:18.4123507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:18.4169285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:19.9197618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:19.9198182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:20.3839882Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:20.3841294Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:20.3842591Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:20.3843379Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:20.9489355Z ok (4.513s)
2022-09-27T16:33:20.9494911Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-09-27T16:33:20.9510778Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46986
2022-09-27T16:33:20.9517049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46987
2022-09-27T16:33:22.6780709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:22.6781263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:22.6781882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:22.6782377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:22.6889202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:22.6889967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:22.6893164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:22.6893665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:22.9487420Z dist init r=1, world=2
2022-09-27T16:33:22.9492828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:22.9601510Z dist init r=0, world=2
2022-09-27T16:33:22.9607801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:22.9608909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:22.9697943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:24.4714323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:24.4714940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:24.9310453Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:24.9311173Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:24.9367435Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:24.9368110Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:24.9427647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:24.9428329Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:24.9429503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:24.9430203Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:25.5618929Z ok (4.613s)
2022-09-27T16:33:25.5623864Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-09-27T16:33:25.5639711Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47071
2022-09-27T16:33:25.5646020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47072
2022-09-27T16:33:27.2400502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:27.2401110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:27.2401683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:27.2402188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:27.3148566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:27.3149177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:27.3150005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:27.3150486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:27.5091310Z dist init r=1, world=2
2022-09-27T16:33:27.5096243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:27.5615626Z dist init r=0, world=2
2022-09-27T16:33:27.5621466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:27.5622548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:27.5705972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:29.0464952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:29.0465544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:29.5188233Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:29.5189185Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:29.5202464Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:29.5203382Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:29.5276155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:29.5276814Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:29.5278026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:29.5278700Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:30.0744377Z ok (4.512s)
2022-09-27T16:33:30.0749114Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-09-27T16:33:30.0765056Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47156
2022-09-27T16:33:30.0771706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47157
2022-09-27T16:33:31.7878179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:31.7878718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:31.7879363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:31.7879838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:31.8106114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:31.8106604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:31.8109908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:31.8110384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:32.0687296Z dist init r=0, world=2
2022-09-27T16:33:32.0687654Z dist init r=1, world=2
2022-09-27T16:33:32.0691511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:32.0693011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:32.0693603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:32.0694281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:33.5623528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:33.5624108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:34.0374455Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:34.0375202Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:34.0431185Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:34.0431853Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:34.0514514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:34.0515159Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:34.0516066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:34.0517026Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:34.5870550Z ok (4.512s)
2022-09-27T16:33:34.5875634Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-09-27T16:33:34.5891487Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47241
2022-09-27T16:33:34.5898479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47242
2022-09-27T16:33:36.2959115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:36.2959660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:36.2960251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:36.2960746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:36.3414858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:36.3415339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:36.3418675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:36.3419162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:36.5700434Z dist init r=1, world=2
2022-09-27T16:33:36.5704308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:36.5949150Z dist init r=0, world=2
2022-09-27T16:33:36.5955318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:36.5956154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:36.6011254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:38.0899295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:38.0900235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:38.5435753Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:38.5436533Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:38.5456191Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:38.5456859Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:38.5558643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:38.5559316Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:38.5560202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:38.5560862Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:39.0999593Z ok (4.513s)
2022-09-27T16:33:39.1004494Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-09-27T16:33:39.1020923Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47326
2022-09-27T16:33:39.1027689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47327
2022-09-27T16:33:40.8324807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:40.8325761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:40.8326449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:40.8326953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:40.8676808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:40.8677294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:40.8679871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:40.8680381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:41.1241448Z dist init r=1, world=2
2022-09-27T16:33:41.1246541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:41.1255296Z dist init r=0, world=2
2022-09-27T16:33:41.1261482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:41.1262432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:41.1349727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:42.6485691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:42.6486223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:43.1065311Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1066172Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:43.1110548Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1111223Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:43.1168790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1169465Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:43.1170644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1171310Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:43.1203872Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1204634Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:43.1205599Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:43.1206318Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:43.7146924Z ok (4.615s)
2022-09-27T16:33:43.7152164Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-09-27T16:33:43.7168095Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47411
2022-09-27T16:33:43.7174947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47412
2022-09-27T16:33:45.4151716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:45.4152258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:45.4152875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:45.4153364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:45.4417621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:45.4418085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:45.4421746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:45.4422256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:45.6941970Z dist init r=0, world=2
2022-09-27T16:33:45.6946652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:45.6987358Z dist init r=1, world=2
2022-09-27T16:33:45.6993176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:45.6994363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:45.7050174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:47.1814482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:47.1815020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:47.6383731Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6384494Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:47.6426204Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6426854Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:47.6498284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6499209Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:47.6500316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6500942Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:47.6531801Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6532441Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:47.6533413Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:47.6534140Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:48.2275007Z ok (4.513s)
2022-09-27T16:33:48.2280181Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-09-27T16:33:48.2296253Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47496
2022-09-27T16:33:48.2302625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47497
2022-09-27T16:33:49.9385643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:49.9386210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:49.9387561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:49.9388299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:49.9559784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:49.9560254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:49.9563688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:49.9564160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:50.2190093Z dist init r=1, world=2
2022-09-27T16:33:50.2194045Z dist init r=0, world=2
2022-09-27T16:33:50.2194462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:50.2200151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:50.2200962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:50.2297721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:51.6979605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:51.6980172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:52.1730481Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1731250Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:52.1732247Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1732946Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:52.1812778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1813472Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:52.1814389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1815033Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:52.1847747Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1848483Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:52.1849450Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:52.1850357Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:52.7406998Z ok (4.513s)
2022-09-27T16:33:52.7412269Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-09-27T16:33:52.7429154Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47581
2022-09-27T16:33:52.7436061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47582
2022-09-27T16:33:54.4715800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:54.4716281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:54.4717234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:54.4717734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:54.4914498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:54.4914957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:54.4917948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:54.4918448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:54.7550223Z dist init r=1, world=2
2022-09-27T16:33:54.7553768Z dist init r=0, world=2
2022-09-27T16:33:54.7554525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:33:54.7559816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:33:54.7561103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:54.7658946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:33:56.2396960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:33:56.2397525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:33:56.7080680Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7081398Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:56.7125647Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7126309Z   in_data = torch.tensor(input[self.rank], device=self.rank)
2022-09-27T16:33:56.7225855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7226590Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:56.7227469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py:1067: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7228128Z   return_norm = torch.tensor(total_norm ** norm_type, device=rank)
2022-09-27T16:33:56.7258743Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7259738Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:56.7260723Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:4295: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
2022-09-27T16:33:56.7261439Z   local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters))
2022-09-27T16:33:57.2537372Z ok (4.513s)
2022-09-27T16:33:57.2537588Z 
2022-09-27T16:33:57.2538030Z ----------------------------------------------------------------------
2022-09-27T16:33:57.2538394Z Ran 16 tests in 74.175s
2022-09-27T16:33:57.2538583Z 
2022-09-27T16:33:57.2538693Z OK
2022-09-27T16:33:57.2538834Z 
2022-09-27T16:33:57.2538952Z Generating XML reports...
2022-09-27T16:33:57.2600747Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220927163243.xml
2022-09-27T16:33:57.2611763Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220927163243.xml
2022-09-27T16:33:57.7142808Z Running distributed/algorithms/quantization/test_quantization ... [2022-09-27 16:33:57.713727]
2022-09-27T16:33:57.7151675Z /usr/bin/mpiexec
2022-09-27T16:33:57.7152498Z MPI not available -- MPI backend tests will be skipped
2022-09-27T16:33:57.7366693Z Running distributed tests for the test backend with env init_method
2022-09-27T16:33:57.7394574Z Running distributed tests for the nccl backend with env init_method
2022-09-27T16:33:57.7404200Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:33:57.739978]
2022-09-27T16:33:57.7407529Z Running distributed tests for the gloo backend with env init_method
2022-09-27T16:33:57.7417800Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:33:57.741321]
2022-09-27T16:33:57.7429752Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:33:57.742532]
2022-09-27T16:33:57.7440010Z Running distributed tests for the ucc backend with env init_method
2022-09-27T16:33:57.7462851Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:33:57.745853]
2022-09-27T16:33:59.7221632Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:33:59.7222747Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7223130Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7223512Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7223881Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7224244Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7224634Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7224989Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:33:59.7253180Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:33:59.7254285Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7254678Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7255051Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7255425Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7255781Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7256176Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:33:59.7256537Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:33:59.7263518Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:33:59.7337135Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:34:01.3095145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:01.3096196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:01.3097929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:01.3098818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:01.4020633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:01.4021535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:01.4023323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:01.4024128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:01.5620559Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:01.5635977Z 
2022-09-27T16:34:01.5636416Z Running tests...
2022-09-27T16:34:01.5636939Z ----------------------------------------------------------------------
2022-09-27T16:34:01.6527875Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:01.6542980Z 
2022-09-27T16:34:01.6543355Z Running tests...
2022-09-27T16:34:01.6543773Z ----------------------------------------------------------------------
2022-09-27T16:34:01.6553045Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-09-27T16:34:01.6553375Z 
2022-09-27T16:34:01.6553866Z ----------------------------------------------------------------------
2022-09-27T16:34:01.6554204Z Ran 1 test in 0.001s
2022-09-27T16:34:01.6554381Z 
2022-09-27T16:34:01.6554495Z OK (skipped=1)
2022-09-27T16:34:01.6554618Z 
2022-09-27T16:34:01.6554746Z Generating XML reports...
2022-09-27T16:34:01.6591466Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163401.xml
2022-09-27T16:34:03.1679500Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:03.2552237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47907
2022-09-27T16:34:03.2556674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47908
2022-09-27T16:34:03.7949115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:03.7949648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:03.7951829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:03.7952329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:04.1421402Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:04.1435373Z 
2022-09-27T16:34:04.1435867Z Running tests...
2022-09-27T16:34:04.1436329Z ----------------------------------------------------------------------
2022-09-27T16:34:04.1444471Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-09-27T16:34:04.1444799Z 
2022-09-27T16:34:04.1445096Z ----------------------------------------------------------------------
2022-09-27T16:34:04.1445408Z Ran 1 test in 0.001s
2022-09-27T16:34:04.1445585Z 
2022-09-27T16:34:04.1445688Z OK (skipped=1)
2022-09-27T16:34:04.1445827Z 
2022-09-27T16:34:04.1445963Z Generating XML reports...
2022-09-27T16:34:04.1480675Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163404.xml
2022-09-27T16:34:04.9112871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:04.9113396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:04.9114245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:04.9114710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:04.9680661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:04.9681113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:04.9684530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:04.9685273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:05.1522738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:05.2001739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:05.2096504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:05.2100904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:05.2105923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:05.2200865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:05.6623718Z ok (4.098s)
2022-09-27T16:34:05.6624006Z 
2022-09-27T16:34:05.6624458Z ----------------------------------------------------------------------
2022-09-27T16:34:05.6624798Z Ran 1 test in 4.099s
2022-09-27T16:34:05.6624957Z 
2022-09-27T16:34:05.6625035Z OK
2022-09-27T16:34:05.6625193Z 
2022-09-27T16:34:05.6625332Z Generating XML reports...
2022-09-27T16:34:05.6662884Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163401.xml
2022-09-27T16:34:06.2178069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:06.2178585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:06.2180565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:06.2181077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:06.4656018Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:06.4670695Z 
2022-09-27T16:34:06.4671223Z Running tests...
2022-09-27T16:34:06.4671747Z ----------------------------------------------------------------------
2022-09-27T16:34:07.8038694Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:07.8039498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:07.8041542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:07.8042045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:08.0605640Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:08.0619750Z 
2022-09-27T16:34:08.0620089Z Running tests...
2022-09-27T16:34:08.0620570Z ----------------------------------------------------------------------
2022-09-27T16:34:08.1274950Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:08.2285507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48051
2022-09-27T16:34:08.2290209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48052
2022-09-27T16:34:09.6950440Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:09.7836365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48116
2022-09-27T16:34:09.7840860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48117
2022-09-27T16:34:09.9366380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:09.9367356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:09.9369262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:09.9370532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:09.9553972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:09.9554909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:09.9558198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:09.9559146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:10.1751733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:10.1962647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:10.1998671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:10.2209909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:10.2211437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:10.2213555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:10.2267281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:10.2270509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:10.2271839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:10.2316951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:11.5174503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:11.5175020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:11.5176066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:11.5176538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:11.5177168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:11.5177664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:11.5178304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:11.5178814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:11.8068368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:11.8088419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:11.8280438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:11.8281014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:11.8281855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:11.8282541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:12.3912246Z ok (4.329s)
2022-09-27T16:34:12.3912507Z 
2022-09-27T16:34:12.3912938Z ----------------------------------------------------------------------
2022-09-27T16:34:12.3913297Z Ran 1 test in 4.329s
2022-09-27T16:34:12.3913480Z 
2022-09-27T16:34:12.3913585Z OK
2022-09-27T16:34:12.3913729Z 
2022-09-27T16:34:12.3914200Z Generating XML reports...
2022-09-27T16:34:12.3951895Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163408.xml
2022-09-27T16:34:13.5419071Z ok (7.074s)
2022-09-27T16:34:13.5419331Z 
2022-09-27T16:34:13.5419768Z ----------------------------------------------------------------------
2022-09-27T16:34:13.5420102Z Ran 1 test in 7.075s
2022-09-27T16:34:13.5420273Z 
2022-09-27T16:34:13.5420372Z OK
2022-09-27T16:34:13.5420511Z 
2022-09-27T16:34:13.5420650Z Generating XML reports...
2022-09-27T16:34:13.5459335Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163406.xml
2022-09-27T16:34:14.6588075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:14.6588635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:14.6590051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:14.6590548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:14.8906351Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:14.8919168Z 
2022-09-27T16:34:14.8919343Z Running tests...
2022-09-27T16:34:14.8920259Z ----------------------------------------------------------------------
2022-09-27T16:34:14.8929147Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-09-27T16:34:14.8929517Z 
2022-09-27T16:34:14.8930243Z ----------------------------------------------------------------------
2022-09-27T16:34:14.8930678Z Ran 1 test in 0.001s
2022-09-27T16:34:14.8930887Z 
2022-09-27T16:34:14.8931030Z OK (skipped=1)
2022-09-27T16:34:14.8931200Z 
2022-09-27T16:34:14.8931271Z Generating XML reports...
2022-09-27T16:34:14.8964248Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163414.xml
2022-09-27T16:34:15.7442822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:15.7443387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:15.7445792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:15.7446300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:16.0444551Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:16.0459195Z 
2022-09-27T16:34:16.0459638Z Running tests...
2022-09-27T16:34:16.0460182Z ----------------------------------------------------------------------
2022-09-27T16:34:16.9125739Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:16.9126410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:16.9127338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:16.9127832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:17.1779295Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:17.1792602Z 
2022-09-27T16:34:17.1792751Z Running tests...
2022-09-27T16:34:17.1793575Z ----------------------------------------------------------------------
2022-09-27T16:34:17.1804208Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-09-27T16:34:17.1804689Z 
2022-09-27T16:34:17.1805459Z ----------------------------------------------------------------------
2022-09-27T16:34:17.1805874Z Ran 1 test in 0.001s
2022-09-27T16:34:17.1806056Z 
2022-09-27T16:34:17.1806164Z OK (skipped=1)
2022-09-27T16:34:17.1806329Z 
2022-09-27T16:34:17.1806466Z Generating XML reports...
2022-09-27T16:34:17.1840319Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163417.xml
2022-09-27T16:34:17.7127235Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:17.8094349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48316
2022-09-27T16:34:17.8099765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48317
2022-09-27T16:34:19.3155629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.3156675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.3157937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.3158870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:19.5571764Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:19.5585718Z 
2022-09-27T16:34:19.5586204Z Running tests...
2022-09-27T16:34:19.5586725Z ----------------------------------------------------------------------
2022-09-27T16:34:19.5596258Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-09-27T16:34:19.5596630Z 
2022-09-27T16:34:19.5596881Z ----------------------------------------------------------------------
2022-09-27T16:34:19.5597287Z Ran 1 test in 0.001s
2022-09-27T16:34:19.5597473Z 
2022-09-27T16:34:19.5597598Z OK (skipped=1)
2022-09-27T16:34:19.5597794Z 
2022-09-27T16:34:19.5597938Z Generating XML reports...
2022-09-27T16:34:19.5634930Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163419.xml
2022-09-27T16:34:19.5684273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.5684754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.5700104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.5700552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.5705096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.5705592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:19.5710724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.5711191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:19.8339152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:19.8356700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:19.8556622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:19.8570413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:19.8571227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:19.8575492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:19.8658184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:19.8660885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:19.8661595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:19.8678254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:21.7924834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:21.7925400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:21.7927045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:21.7927817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:22.0791517Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:22.0805165Z 
2022-09-27T16:34:22.0805490Z Running tests...
2022-09-27T16:34:22.0806427Z ----------------------------------------------------------------------
2022-09-27T16:34:22.0817202Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-09-27T16:34:22.0817971Z 
2022-09-27T16:34:22.0818516Z ----------------------------------------------------------------------
2022-09-27T16:34:22.0818887Z Ran 1 test in 0.001s
2022-09-27T16:34:22.0819069Z 
2022-09-27T16:34:22.0819162Z OK (skipped=1)
2022-09-27T16:34:22.0819837Z 
2022-09-27T16:34:22.0820103Z Generating XML reports...
2022-09-27T16:34:22.0855254Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163422.xml
2022-09-27T16:34:23.2231404Z ok (7.177s)
2022-09-27T16:34:23.2231823Z 
2022-09-27T16:34:23.2232605Z ----------------------------------------------------------------------
2022-09-27T16:34:23.2232984Z Ran 1 test in 7.177s
2022-09-27T16:34:23.2233160Z 
2022-09-27T16:34:23.2233287Z OK
2022-09-27T16:34:23.2233742Z 
2022-09-27T16:34:23.2233861Z Generating XML reports...
2022-09-27T16:34:23.2273421Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163416.xml
2022-09-27T16:34:25.3132087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:25.3132741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:25.3135293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:25.3135811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:25.5519247Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:25.5533343Z 
2022-09-27T16:34:25.5533634Z Running tests...
2022-09-27T16:34:25.5534314Z ----------------------------------------------------------------------
2022-09-27T16:34:27.1351233Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:27.2205854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48504
2022-09-27T16:34:27.2212213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48505
2022-09-27T16:34:28.8557547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:28.8558096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:28.8559492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:28.8560285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:28.8824432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:28.8824949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:28.8828894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:28.8829375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:29.1153667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:29.1285983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:29.1367872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:29.1501019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:29.1501918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:29.1504933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:29.1573099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:29.1575771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:29.1576461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:29.1607477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:32.4345050Z ok (6.881s)
2022-09-27T16:34:32.4345577Z 
2022-09-27T16:34:32.4346295Z ----------------------------------------------------------------------
2022-09-27T16:34:32.4346692Z Ran 1 test in 6.881s
2022-09-27T16:34:32.4346871Z 
2022-09-27T16:34:32.4346948Z OK
2022-09-27T16:34:32.4347091Z 
2022-09-27T16:34:32.4347556Z Generating XML reports...
2022-09-27T16:34:32.4384822Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163425.xml
2022-09-27T16:34:34.5170925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:34.5171507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:34.5173979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:34.5174544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:34.7587075Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:34.7600300Z 
2022-09-27T16:34:34.7600561Z Running tests...
2022-09-27T16:34:34.7601298Z ----------------------------------------------------------------------
2022-09-27T16:34:36.3891092Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:36.4743906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48627
2022-09-27T16:34:36.4748470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48628
2022-09-27T16:34:38.1676698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:38.1677245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:38.1679287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:38.1679826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:38.1790271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:38.1790791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:38.1794124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:38.1794601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:38.4229472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:38.4355196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:38.4441120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:38.4578851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:38.4579674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:38.4582790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:38.4647383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:38.4650011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:38.4650983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:38.4685794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:41.6874537Z ok (6.927s)
2022-09-27T16:34:41.6874764Z 
2022-09-27T16:34:41.6875203Z ----------------------------------------------------------------------
2022-09-27T16:34:41.6875531Z Ran 1 test in 6.927s
2022-09-27T16:34:41.6875765Z 
2022-09-27T16:34:41.6875836Z OK
2022-09-27T16:34:41.6875947Z 
2022-09-27T16:34:41.6876082Z Generating XML reports...
2022-09-27T16:34:41.6914500Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163434.xml
2022-09-27T16:34:42.4190334Z Running distributed tests for the test backend with file init_method
2022-09-27T16:34:42.4210855Z Running distributed tests for the nccl backend with file init_method
2022-09-27T16:34:42.4220563Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:34:42.421617]
2022-09-27T16:34:42.4222574Z Running distributed tests for the gloo backend with file init_method
2022-09-27T16:34:42.4232001Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:34:42.422753]
2022-09-27T16:34:42.4244062Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:34:42.423964]
2022-09-27T16:34:42.4253800Z Running distributed tests for the ucc backend with file init_method
2022-09-27T16:34:42.4275668Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:34:42.427138]
2022-09-27T16:34:44.4110744Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:34:44.4118047Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:34:44.4119091Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4119477Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4119888Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4120292Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4120673Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4121104Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4121485Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:34:44.4199998Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-09-27T16:34:44.4200985Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4201359Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4201748Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4202127Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4202492Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4202886Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-09-27T16:34:44.4203247Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:34:44.4215722Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:34:46.0635165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:46.0635723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:46.0638130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:46.0638940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:46.1047198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:46.1047654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:46.1051358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:46.1051867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:46.3230468Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:46.3244082Z 
2022-09-27T16:34:46.3244509Z Running tests...
2022-09-27T16:34:46.3245068Z ----------------------------------------------------------------------
2022-09-27T16:34:46.3511884Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:46.3527689Z 
2022-09-27T16:34:46.3528001Z Running tests...
2022-09-27T16:34:46.3528455Z ----------------------------------------------------------------------
2022-09-27T16:34:46.3537121Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-09-27T16:34:46.3537448Z 
2022-09-27T16:34:46.3537745Z ----------------------------------------------------------------------
2022-09-27T16:34:46.3538070Z Ran 1 test in 0.001s
2022-09-27T16:34:46.3538242Z 
2022-09-27T16:34:46.3538360Z OK (skipped=1)
2022-09-27T16:34:46.3538525Z 
2022-09-27T16:34:46.3538655Z Generating XML reports...
2022-09-27T16:34:46.3574084Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163446.xml
2022-09-27T16:34:47.9238036Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:48.0198779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48955
2022-09-27T16:34:48.0203798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48956
2022-09-27T16:34:48.4680749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:48.4681316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:48.4683918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:48.4684438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:48.7964402Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:48.7994378Z 
2022-09-27T16:34:48.7995758Z Running tests...
2022-09-27T16:34:48.7997218Z ----------------------------------------------------------------------
2022-09-27T16:34:48.8014520Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.002s)
2022-09-27T16:34:48.8015231Z 
2022-09-27T16:34:48.8015884Z ----------------------------------------------------------------------
2022-09-27T16:34:48.8016610Z Ran 1 test in 0.002s
2022-09-27T16:34:48.8016970Z 
2022-09-27T16:34:48.8017193Z OK (skipped=1)
2022-09-27T16:34:48.8017528Z 
2022-09-27T16:34:48.8017798Z Generating XML reports...
2022-09-27T16:34:48.8085239Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163448.xml
2022-09-27T16:34:49.7211842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:49.7212482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:49.7215358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:49.7216239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:49.7583797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:49.7584449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:49.7587688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:49.7588215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:50.0151660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:50.0501647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:50.0612062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:50.0613235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:50.0614125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:50.0614832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:50.5272388Z ok (4.202s)
2022-09-27T16:34:50.5272882Z 
2022-09-27T16:34:50.5273552Z ----------------------------------------------------------------------
2022-09-27T16:34:50.5273922Z Ran 1 test in 4.203s
2022-09-27T16:34:50.5274100Z 
2022-09-27T16:34:50.5274202Z OK
2022-09-27T16:34:50.5274346Z 
2022-09-27T16:34:50.5274489Z Generating XML reports...
2022-09-27T16:34:50.5311856Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163446.xml
2022-09-27T16:34:50.8810618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:50.8811689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:50.8812780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:50.8813299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:51.1200950Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:51.1215125Z 
2022-09-27T16:34:51.1215680Z Running tests...
2022-09-27T16:34:51.1216163Z ----------------------------------------------------------------------
2022-09-27T16:34:52.7778125Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:52.7778762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:52.7780167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:52.7780672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:52.8044442Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:52.9168268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49098
2022-09-27T16:34:52.9173699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49099
2022-09-27T16:34:53.0331191Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:34:53.0346213Z 
2022-09-27T16:34:53.0346708Z Running tests...
2022-09-27T16:34:53.0347224Z ----------------------------------------------------------------------
2022-09-27T16:34:54.6290718Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:54.6291652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:54.6293843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:54.6294322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:54.6494529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:54.6495019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:54.6498990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:54.6499465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:54.7571289Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:54.8542484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49166
2022-09-27T16:34:54.8546691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49167
2022-09-27T16:34:54.8728267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:54.8944790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:54.8984247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:54.9199526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:54.9201167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:54.9202398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:54.9249495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:54.9253635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:54.9254336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:54.9305213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:56.6365705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:56.6366261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:56.6367600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:56.6368118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:56.6469663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:56.6470165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:56.6474739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:56.6475237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:56.9260883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:56.9288262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:56.9472047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:56.9472607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:56.9473510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:56.9474522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:57.4618694Z ok (4.427s)
2022-09-27T16:34:57.4618934Z 
2022-09-27T16:34:57.4619362Z ----------------------------------------------------------------------
2022-09-27T16:34:57.4619695Z Ran 1 test in 4.427s
2022-09-27T16:34:57.4619876Z 
2022-09-27T16:34:57.4619974Z OK
2022-09-27T16:34:57.4620115Z 
2022-09-27T16:34:57.4620254Z Generating XML reports...
2022-09-27T16:34:57.4659763Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163453.xml
2022-09-27T16:34:58.3311177Z ok (7.209s)
2022-09-27T16:34:58.3311470Z 
2022-09-27T16:34:58.3311927Z ----------------------------------------------------------------------
2022-09-27T16:34:58.3312283Z Ran 1 test in 7.209s
2022-09-27T16:34:58.3312459Z 
2022-09-27T16:34:58.3312559Z OK
2022-09-27T16:34:58.3312676Z 
2022-09-27T16:34:58.3312825Z Generating XML reports...
2022-09-27T16:34:58.3352780Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163451.xml
2022-09-27T16:34:59.8167011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:59.8167501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:59.8168770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:59.8169268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:00.0571577Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:00.0586815Z 
2022-09-27T16:35:00.0586964Z Running tests...
2022-09-27T16:35:00.0587408Z ----------------------------------------------------------------------
2022-09-27T16:35:00.0597049Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-09-27T16:35:00.0597373Z 
2022-09-27T16:35:00.0597660Z ----------------------------------------------------------------------
2022-09-27T16:35:00.0597974Z Ran 1 test in 0.001s
2022-09-27T16:35:00.0598140Z 
2022-09-27T16:35:00.0598253Z OK (skipped=1)
2022-09-27T16:35:00.0598411Z 
2022-09-27T16:35:00.0598540Z Generating XML reports...
2022-09-27T16:35:00.0633050Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163500.xml
2022-09-27T16:35:00.5951957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:00.5952515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:00.5954182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:00.5954680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:00.8309713Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:00.8323025Z 
2022-09-27T16:35:00.8323164Z Running tests...
2022-09-27T16:35:00.8324098Z ----------------------------------------------------------------------
2022-09-27T16:35:02.1405840Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:02.1406979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:02.1408576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:02.1409409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:02.4166986Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:02.4181019Z 
2022-09-27T16:35:02.4181173Z Running tests...
2022-09-27T16:35:02.4182056Z ----------------------------------------------------------------------
2022-09-27T16:35:02.4191478Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-09-27T16:35:02.4192014Z 
2022-09-27T16:35:02.4192635Z ----------------------------------------------------------------------
2022-09-27T16:35:02.4193112Z Ran 1 test in 0.001s
2022-09-27T16:35:02.4193285Z 
2022-09-27T16:35:02.4193399Z OK (skipped=1)
2022-09-27T16:35:02.4193559Z 
2022-09-27T16:35:02.4193763Z Generating XML reports...
2022-09-27T16:35:02.4229350Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163502.xml
2022-09-27T16:35:02.5412714Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:02.6476933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49363
2022-09-27T16:35:02.6481149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49364
2022-09-27T16:35:04.3664099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:04.3665104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:04.3666308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:04.3667279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:04.4483354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:04.4484899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:04.4486170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:04.4487216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:04.5645452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:04.5646467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:04.5647664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:04.5648548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:04.6023722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:04.6238634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:04.7335664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:04.7549359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:04.7551032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:04.7552587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:35:04.7556689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:04.7560457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:35:04.7561866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:04.7656352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:04.8066067Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:04.8081358Z 
2022-09-27T16:35:04.8081767Z Running tests...
2022-09-27T16:35:04.8082636Z ----------------------------------------------------------------------
2022-09-27T16:35:04.8094092Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-09-27T16:35:04.8094741Z 
2022-09-27T16:35:04.8095362Z ----------------------------------------------------------------------
2022-09-27T16:35:04.8095975Z Ran 1 test in 0.001s
2022-09-27T16:35:04.8096286Z 
2022-09-27T16:35:04.8096481Z OK (skipped=1)
2022-09-27T16:35:04.8096781Z 
2022-09-27T16:35:04.8096990Z Generating XML reports...
2022-09-27T16:35:04.8132598Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163504.xml
2022-09-27T16:35:07.0669266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:07.0670270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:07.0671414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:07.0672344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:07.3438333Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:07.3451614Z 
2022-09-27T16:35:07.3452087Z Running tests...
2022-09-27T16:35:07.3452674Z ----------------------------------------------------------------------
2022-09-27T16:35:07.3463068Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-09-27T16:35:07.3463423Z 
2022-09-27T16:35:07.3463715Z ----------------------------------------------------------------------
2022-09-27T16:35:07.3464058Z Ran 1 test in 0.001s
2022-09-27T16:35:07.3464236Z 
2022-09-27T16:35:07.3464350Z OK (skipped=1)
2022-09-27T16:35:07.3464514Z 
2022-09-27T16:35:07.3464642Z Generating XML reports...
2022-09-27T16:35:07.3500585Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163507.xml
2022-09-27T16:35:08.1625151Z ok (7.330s)
2022-09-27T16:35:08.1625469Z 
2022-09-27T16:35:08.1625909Z ----------------------------------------------------------------------
2022-09-27T16:35:08.1626266Z Ran 1 test in 7.330s
2022-09-27T16:35:08.1626444Z 
2022-09-27T16:35:08.1626544Z OK
2022-09-27T16:35:08.1626659Z 
2022-09-27T16:35:08.1626804Z Generating XML reports...
2022-09-27T16:35:08.1664574Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163500.xml
2022-09-27T16:35:10.3436721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:10.3437312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:10.3438781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:10.3439282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:10.5895575Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:10.5908542Z 
2022-09-27T16:35:10.5908978Z Running tests...
2022-09-27T16:35:10.5909503Z ----------------------------------------------------------------------
2022-09-27T16:35:12.1790887Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:12.2669545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49552
2022-09-27T16:35:12.2675384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49553
2022-09-27T16:35:13.9292275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:13.9292822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:13.9294000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:13.9294474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:13.9425448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:13.9425932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:13.9429911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:13.9430386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:14.1913830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:14.1926344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:14.2127248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:14.2138506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:14.2139343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:14.2142785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:35:14.2232154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:14.2234676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:35:14.2235386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:14.2245376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:17.4826955Z ok (6.891s)
2022-09-27T16:35:17.4827166Z 
2022-09-27T16:35:17.4827631Z ----------------------------------------------------------------------
2022-09-27T16:35:17.4828003Z Ran 1 test in 6.892s
2022-09-27T16:35:17.4828190Z 
2022-09-27T16:35:17.4828304Z OK
2022-09-27T16:35:17.4828426Z 
2022-09-27T16:35:17.4828576Z Generating XML reports...
2022-09-27T16:35:17.4869245Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163510.xml
2022-09-27T16:35:19.6247996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:19.6248572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:19.6250120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:19.6250611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:19.8654608Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-09-27T16:35:19.8667992Z 
2022-09-27T16:35:19.8668297Z Running tests...
2022-09-27T16:35:19.8668747Z ----------------------------------------------------------------------
2022-09-27T16:35:21.4782468Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:21.5619355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49675
2022-09-27T16:35:21.5625106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49676
2022-09-27T16:35:23.1935782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:23.1936704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:23.1937627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:23.1938083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:23.2231305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:23.2231759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:23.2235647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:23.2236147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:23.4715874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:23.4912158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:23.4930925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:23.5125081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:23.5125914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:23.5128821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:35:23.5136034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:23.5139306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:35:23.5140272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:23.5231849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:26.7748345Z ok (6.908s)
2022-09-27T16:35:26.7748606Z 
2022-09-27T16:35:26.7749227Z ----------------------------------------------------------------------
2022-09-27T16:35:26.7749683Z Ran 1 test in 6.908s
2022-09-27T16:35:26.7749998Z 
2022-09-27T16:35:26.7750144Z OK
2022-09-27T16:35:26.7750387Z 
2022-09-27T16:35:26.7750633Z Generating XML reports...
2022-09-27T16:35:26.7787726Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163519.xml
2022-09-27T16:35:27.5653048Z Running distributed/test_pg_wrapper ... [2022-09-27 16:35:27.564688]
2022-09-27T16:35:27.5653851Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:35:27.564787]
2022-09-27T16:35:29.4571646Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:35:29.4572255Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:35:29.4574443Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-09-27T16:35:29.4577349Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4577810Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4578272Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4578756Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4579258Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4579680Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4580131Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4580595Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4581087Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-09-27T16:35:29.4582753Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-09-27T16:35:29.4584770Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:35:29.4585213Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:35:29.4585662Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:35:29.4586126Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:35:29.4586759Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-09-27T16:35:31.0733824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:31.0734318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:31.0735616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:31.0736115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:31.3445211Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:35:31.3460903Z 
2022-09-27T16:35:31.3461403Z Running tests...
2022-09-27T16:35:31.3461882Z ----------------------------------------------------------------------
2022-09-27T16:35:32.9897680Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:33.1207510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49831
2022-09-27T16:35:33.1212968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49832
2022-09-27T16:35:33.1220051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49833
2022-09-27T16:35:33.1226192Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49834
2022-09-27T16:35:34.8177486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.8178011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.8179052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.8179519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:34.8334113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.8334589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.8337878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.8338695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:34.8446273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.8446746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.8450581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.8451074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:34.9028584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.9029193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.9031426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.9031942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:35.1108411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:35:35.1206816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:35:35.1262452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:35.1664008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:35.1823338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:35.1979418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:35:35.1979988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:35.1980752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:35:35.1981534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:35.1982243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:35.1982929Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:35.2027586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:35.2856440Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:35:35.2856999Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:35:35.2955416Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:35:35.3056187Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-09-27T16:35:35.9305882Z ok (4.584s)
2022-09-27T16:35:35.9306167Z 
2022-09-27T16:35:35.9306620Z ----------------------------------------------------------------------
2022-09-27T16:35:35.9306950Z Ran 1 test in 4.584s
2022-09-27T16:35:35.9307126Z 
2022-09-27T16:35:35.9307223Z OK
2022-09-27T16:35:35.9307340Z 
2022-09-27T16:35:35.9307479Z Generating XML reports...
2022-09-27T16:35:35.9346716Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163531.xml
2022-09-27T16:35:38.0555661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:38.0556524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:38.0557351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:38.0557795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:38.3180705Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:35:38.3194692Z 
2022-09-27T16:35:38.3195116Z Running tests...
2022-09-27T16:35:38.3195582Z ----------------------------------------------------------------------
2022-09-27T16:35:39.9563286Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:40.0371365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50038
2022-09-27T16:35:40.0376549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50039
2022-09-27T16:35:40.0383362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50040
2022-09-27T16:35:40.0390411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50041
2022-09-27T16:35:41.7340653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:41.7341685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:41.7342855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:41.7343759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:41.7344832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:41.7346217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:41.7347453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:41.7348408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:41.7611135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:41.7612027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:41.7616758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:41.7617719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:41.8063169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:41.8064168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:41.8066361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:41.8067300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:42.0475721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:42.0484568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:35:42.0501615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:35:42.0736628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:42.1092617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:42.1195046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:35:42.1195594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:42.1196080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:35:42.1197235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:42.1197972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:42.1198678Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:42.1297429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:35:42.8469642Z ok (4.527s)
2022-09-27T16:35:42.8469897Z 
2022-09-27T16:35:42.8470339Z ----------------------------------------------------------------------
2022-09-27T16:35:42.8470690Z Ran 1 test in 4.527s
2022-09-27T16:35:42.8470864Z 
2022-09-27T16:35:42.8470966Z OK
2022-09-27T16:35:42.8471107Z 
2022-09-27T16:35:42.8471263Z Generating XML reports...
2022-09-27T16:35:42.8510883Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163538.xml
2022-09-27T16:35:44.9680546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:44.9681094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:44.9681692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:44.9682198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:45.2301181Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:35:45.2314447Z 
2022-09-27T16:35:45.2315346Z Running tests...
2022-09-27T16:35:45.2315889Z ----------------------------------------------------------------------
2022-09-27T16:35:46.8514205Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:46.9353829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50245
2022-09-27T16:35:46.9357848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50246
2022-09-27T16:35:46.9364561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50247
2022-09-27T16:35:46.9371505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50248
2022-09-27T16:35:48.6172631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:48.6173613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:48.6174813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:48.6175661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:48.6176815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:48.6177742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:48.6178899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:48.6179847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:48.6181024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:48.6181949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:48.6183073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:48.6184010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:48.6224595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:48.6225950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:48.6228538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:48.6229296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:48.9122393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:35:48.9158762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:48.9193239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:35:48.9197921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:49.5446948Z skip: Need at least 4 CUDA devices (4.313s)
2022-09-27T16:35:49.5447459Z 
2022-09-27T16:35:49.5448135Z ----------------------------------------------------------------------
2022-09-27T16:35:49.5448682Z Ran 1 test in 4.313s
2022-09-27T16:35:49.5448954Z 
2022-09-27T16:35:49.5449180Z OK (skipped=1)
2022-09-27T16:35:49.5449523Z 
2022-09-27T16:35:49.5450159Z Generating XML reports...
2022-09-27T16:35:49.5491569Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163545.xml
2022-09-27T16:35:51.6858920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:51.6859480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:51.6860569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:51.6861412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:51.9511217Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:35:51.9525025Z 
2022-09-27T16:35:51.9525180Z Running tests...
2022-09-27T16:35:51.9526096Z ----------------------------------------------------------------------
2022-09-27T16:35:53.6142176Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:53.6978091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50416
2022-09-27T16:35:53.6983619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50417
2022-09-27T16:35:53.6990156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50418
2022-09-27T16:35:53.6996791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50419
2022-09-27T16:35:55.3913740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.3914721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.3915825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.3916669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.3927266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.3928210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.3933890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.3934817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.3978711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.3979631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.3982458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.3983369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.4112174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.4113128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.4116427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.4117400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.6815347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:35:55.6827877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:35:55.6990936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:55.7033695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:56.3070150Z skip: Need at least 4 CUDA devices (4.354s)
2022-09-27T16:35:56.3070416Z 
2022-09-27T16:35:56.3070854Z ----------------------------------------------------------------------
2022-09-27T16:35:56.3071206Z Ran 1 test in 4.354s
2022-09-27T16:35:56.3071377Z 
2022-09-27T16:35:56.3071492Z OK (skipped=1)
2022-09-27T16:35:56.3071657Z 
2022-09-27T16:35:56.3071765Z Generating XML reports...
2022-09-27T16:35:56.3112513Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163551.xml
2022-09-27T16:35:58.3994104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:58.3994702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:58.3995600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:58.3996076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:58.6589048Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:35:58.6602410Z 
2022-09-27T16:35:58.6602636Z Running tests...
2022-09-27T16:35:58.6603107Z ----------------------------------------------------------------------
2022-09-27T16:36:00.2743668Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:00.3619912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50587
2022-09-27T16:36:00.3624080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50588
2022-09-27T16:36:00.3630918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50589
2022-09-27T16:36:00.3637747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50590
2022-09-27T16:36:02.0608455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:02.0609383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:02.0610309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:02.0610745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:02.0611358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:02.0611819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:02.0614315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:02.0614788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:02.0923084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:02.0923573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:02.0927353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:02.0927843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:02.1397754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:02.1398238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:02.1401192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:02.1401682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:02.3454224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:02.3480323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:02.3634194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:36:02.4075172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:36:02.4785202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:02.4887308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:02.4989702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:36:02.4990450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:36:02.4991296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:02.4992077Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:02.4992800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:02.5092158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:02.5610884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:36:02.5815134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:36:02.5917906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-09-27T16:36:02.5918389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:36:02.5919125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:02.5919813Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:02.5920510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:02.6016835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:03.3728861Z ok (4.712s)
2022-09-27T16:36:03.3729156Z 
2022-09-27T16:36:03.3729828Z ----------------------------------------------------------------------
2022-09-27T16:36:03.3730249Z Ran 1 test in 4.713s
2022-09-27T16:36:03.3730430Z 
2022-09-27T16:36:03.3730531Z OK
2022-09-27T16:36:03.3730680Z 
2022-09-27T16:36:03.3730801Z Generating XML reports...
2022-09-27T16:36:03.3769992Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163558.xml
2022-09-27T16:36:05.5104527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:05.5105039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:05.5106487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:05.5106982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:05.7767618Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:05.7782366Z 
2022-09-27T16:36:05.7782488Z Running tests...
2022-09-27T16:36:05.7783390Z ----------------------------------------------------------------------
2022-09-27T16:36:07.4217256Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:07.5105971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50806
2022-09-27T16:36:07.5110714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50807
2022-09-27T16:36:07.5117698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50808
2022-09-27T16:36:07.5125185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50809
2022-09-27T16:36:09.2044655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:09.2045207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:09.2046154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:09.2046621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:09.2050655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:09.2051609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:09.2052796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:09.2053692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:09.2076343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:09.2076820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:09.2080297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:09.2080773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:09.2372609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:09.2373092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:09.2377317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:09.2377802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:09.4977918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:09.5159862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:36:09.5178028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:36:09.5305251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:09.5593780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:09.5797424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:09.5900501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:36:09.5900989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:36:09.5902002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:09.5902855Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:09.5903671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:09.6000657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:10.4206299Z ok (4.642s)
2022-09-27T16:36:10.4206715Z 
2022-09-27T16:36:10.4207424Z ----------------------------------------------------------------------
2022-09-27T16:36:10.4207794Z Ran 1 test in 4.642s
2022-09-27T16:36:10.4207972Z 
2022-09-27T16:36:10.4208093Z OK
2022-09-27T16:36:10.4208265Z 
2022-09-27T16:36:10.4208351Z Generating XML reports...
2022-09-27T16:36:10.4247300Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163605.xml
2022-09-27T16:36:12.5612276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:12.5612761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:12.5614143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:12.5614669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:12.8377078Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:12.8391651Z 
2022-09-27T16:36:12.8391837Z Running tests...
2022-09-27T16:36:12.8392580Z ----------------------------------------------------------------------
2022-09-27T16:36:14.4789605Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:14.5728458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51013
2022-09-27T16:36:14.5733539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51014
2022-09-27T16:36:14.5740531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51015
2022-09-27T16:36:14.5747537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51016
2022-09-27T16:36:16.2833326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.2833887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.2834681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.2835257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.2835858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.2836320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.2837992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.2838473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.3506357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.3506818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.3509613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.3510107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.3540679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.3541158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.3545271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.3545760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.5872537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:36:16.5877906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:16.6168714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:36:16.6188165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:17.1829872Z skip: Need at least 4 CUDA devices (4.343s)
2022-09-27T16:36:17.1830148Z 
2022-09-27T16:36:17.1830547Z ----------------------------------------------------------------------
2022-09-27T16:36:17.1830930Z Ran 1 test in 4.344s
2022-09-27T16:36:17.1831079Z 
2022-09-27T16:36:17.1831172Z OK (skipped=1)
2022-09-27T16:36:17.1831335Z 
2022-09-27T16:36:17.1831475Z Generating XML reports...
2022-09-27T16:36:17.1872135Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163612.xml
2022-09-27T16:36:19.3498590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:19.3499253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:19.3500136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:19.3500751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:19.6168484Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:19.6183460Z 
2022-09-27T16:36:19.6183679Z Running tests...
2022-09-27T16:36:19.6184120Z ----------------------------------------------------------------------
2022-09-27T16:36:21.2721304Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:21.3645233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51184
2022-09-27T16:36:21.3650107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51185
2022-09-27T16:36:21.3658000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51186
2022-09-27T16:36:21.3664696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51187
2022-09-27T16:36:23.0673244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:23.0674337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:23.0675457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:23.0676353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:23.0677487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:23.0678388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:23.0679595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:23.0680588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:23.0769111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:23.0769986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:23.0773696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:23.0774225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:23.0815609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:23.0816070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:23.0819956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:23.0820435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:23.3668335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:23.3722298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:36:23.3798101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:23.3818267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:36:23.9738250Z skip: Need at least 4 CUDA devices (4.355s)
2022-09-27T16:36:23.9738522Z 
2022-09-27T16:36:23.9738960Z ----------------------------------------------------------------------
2022-09-27T16:36:23.9739283Z Ran 1 test in 4.355s
2022-09-27T16:36:23.9739451Z 
2022-09-27T16:36:23.9739568Z OK (skipped=1)
2022-09-27T16:36:23.9739729Z 
2022-09-27T16:36:23.9740171Z Generating XML reports...
2022-09-27T16:36:23.9778670Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163619.xml
2022-09-27T16:36:26.0663721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:26.0664639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:26.0665243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:26.0665738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:26.3219790Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:26.3232725Z 
2022-09-27T16:36:26.3233151Z Running tests...
2022-09-27T16:36:26.3233752Z ----------------------------------------------------------------------
2022-09-27T16:36:27.9499540Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:28.0357843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51355
2022-09-27T16:36:28.0362166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51356
2022-09-27T16:36:28.0368423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51357
2022-09-27T16:36:28.0375909Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51358
2022-09-27T16:36:29.7447873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:29.7448924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:29.7450387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:29.7451402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:29.7472601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:29.7473951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:29.7477421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:29.7478351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:29.7479559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:29.7480446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:29.7483672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:29.7484650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:29.7579330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:29.7580220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:29.7583487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:29.7584365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:30.0387594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:36:30.0388655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:30.0389530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:36:30.0470695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:30.0912595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:30.1116710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:30.1219334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:36:30.1220316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:36:30.1221743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:30.1223026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:30.1223719Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:30.1319079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:36:30.2040981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:36:30.2143146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:36:30.2144127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-09-27T16:36:30.2145072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-09-27T16:36:30.2146673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:30.2148033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:30.2149446Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:30.2150839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-09-27T16:36:30.9467009Z ok (4.623s)
2022-09-27T16:36:30.9467681Z 
2022-09-27T16:36:30.9468308Z ----------------------------------------------------------------------
2022-09-27T16:36:30.9468677Z Ran 1 test in 4.623s
2022-09-27T16:36:30.9468857Z 
2022-09-27T16:36:30.9468932Z OK
2022-09-27T16:36:30.9469077Z 
2022-09-27T16:36:30.9469279Z Generating XML reports...
2022-09-27T16:36:30.9506511Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163626.xml
2022-09-27T16:36:33.0808965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:33.0810017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:33.0810722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:33.0811223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:33.3434167Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:33.3447491Z 
2022-09-27T16:36:33.3447687Z Running tests...
2022-09-27T16:36:33.3448192Z ----------------------------------------------------------------------
2022-09-27T16:36:34.9830006Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:35.0679603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51574
2022-09-27T16:36:35.0683936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51575
2022-09-27T16:36:36.7089132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:36.7090152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:36.7091160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:36.7091853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:36.7400641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:36.7401128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:36.7404757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:36.7405450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:36.9815807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:37.0014057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:37.0030682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:37.0229940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:37.0231444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:37.0235198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:37.0544557Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:36:37.0545135Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:36:37.4744891Z ok (4.129s)
2022-09-27T16:36:37.4745178Z 
2022-09-27T16:36:37.4745883Z ----------------------------------------------------------------------
2022-09-27T16:36:37.4746486Z Ran 1 test in 4.130s
2022-09-27T16:36:37.4746740Z 
2022-09-27T16:36:37.4746868Z OK
2022-09-27T16:36:37.4747126Z 
2022-09-27T16:36:37.4747377Z Generating XML reports...
2022-09-27T16:36:37.4788953Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163633.xml
2022-09-27T16:36:39.5597552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:39.5598105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:39.5599218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:39.5599718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:39.8204728Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:39.8219263Z 
2022-09-27T16:36:39.8219537Z Running tests...
2022-09-27T16:36:39.8220017Z ----------------------------------------------------------------------
2022-09-27T16:36:41.4184547Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:41.4986445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51691
2022-09-27T16:36:41.4991546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51692
2022-09-27T16:36:43.1618978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:43.1619520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:43.1620679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:43.1621172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:43.2375803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:43.2376608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:43.2379825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:43.2380341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:43.4231305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:43.4440862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:43.4901320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:43.5115019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:43.5115836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:43.5150802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:45.7090682Z ok (5.887s)
2022-09-27T16:36:45.7090949Z 
2022-09-27T16:36:45.7091413Z ----------------------------------------------------------------------
2022-09-27T16:36:45.7091779Z Ran 1 test in 5.887s
2022-09-27T16:36:45.7091897Z 
2022-09-27T16:36:45.7091999Z OK
2022-09-27T16:36:45.7092116Z 
2022-09-27T16:36:45.7092259Z Generating XML reports...
2022-09-27T16:36:45.7131986Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163639.xml
2022-09-27T16:36:47.8007152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:47.8007830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:47.8008927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:47.8009381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:48.0665522Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:48.0680024Z 
2022-09-27T16:36:48.0680241Z Running tests...
2022-09-27T16:36:48.0680764Z ----------------------------------------------------------------------
2022-09-27T16:36:49.6723834Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:49.7524898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51824
2022-09-27T16:36:49.7530021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51825
2022-09-27T16:36:51.4216365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.4216888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.4217949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.4218457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.4466188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.4466684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.4470441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.4470926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.6898677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:51.7029641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:51.7451464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:51.7452039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:51.7452846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:51.7453568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:51.7561430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:36:51.7561941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:36:51.7562618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:36:51.7563323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:36:53.9628691Z ok (5.894s)
2022-09-27T16:36:53.9628902Z 
2022-09-27T16:36:53.9629343Z ----------------------------------------------------------------------
2022-09-27T16:36:53.9629737Z Ran 1 test in 5.895s
2022-09-27T16:36:53.9629917Z 
2022-09-27T16:36:53.9630014Z OK
2022-09-27T16:36:53.9630165Z 
2022-09-27T16:36:53.9630268Z Generating XML reports...
2022-09-27T16:36:53.9669979Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163648.xml
2022-09-27T16:36:56.0704511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:56.0705058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:56.0706285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:56.0706804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:56.3365883Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:36:56.3380992Z 
2022-09-27T16:36:56.3381557Z Running tests...
2022-09-27T16:36:56.3381979Z ----------------------------------------------------------------------
2022-09-27T16:36:57.9297310Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:58.0094793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51963
2022-09-27T16:36:58.0100332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51964
2022-09-27T16:36:59.7198798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:59.7199302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:59.7200175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:59.7200666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:59.7695970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:59.7696454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:59.7700470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:59.7700905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:59.9818116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:00.0027652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:00.0223326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:00.0434878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:00.0435722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:00.0535075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:03.4223548Z ok (7.084s)
2022-09-27T16:37:03.4223753Z 
2022-09-27T16:37:03.4224206Z ----------------------------------------------------------------------
2022-09-27T16:37:03.4224562Z Ran 1 test in 7.084s
2022-09-27T16:37:03.4224744Z 
2022-09-27T16:37:03.4224837Z OK
2022-09-27T16:37:03.4224976Z 
2022-09-27T16:37:03.4225093Z Generating XML reports...
2022-09-27T16:37:03.4264570Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163656.xml
2022-09-27T16:37:05.4783698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:05.4796465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:05.4797442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:05.4797928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:05.7350434Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-09-27T16:37:05.7363714Z 
2022-09-27T16:37:05.7363858Z Running tests...
2022-09-27T16:37:05.7364447Z ----------------------------------------------------------------------
2022-09-27T16:37:07.3057021Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:07.3852912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52097
2022-09-27T16:37:07.3858149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52098
2022-09-27T16:37:09.0620217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:09.0621192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:09.0622205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:09.0622696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:09.0816659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:09.0817150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:09.0820978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:09.0821472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:09.3349103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:09.3466797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:09.3869649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:09.3870198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:09.3871011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:09.3871692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:09.4079091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:37:09.4079841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:37:09.4080557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:09.4081416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:12.7981339Z ok (7.061s)
2022-09-27T16:37:12.7982042Z 
2022-09-27T16:37:12.7982563Z ----------------------------------------------------------------------
2022-09-27T16:37:12.7982862Z Ran 1 test in 7.062s
2022-09-27T16:37:12.7983037Z 
2022-09-27T16:37:12.7983137Z OK
2022-09-27T16:37:12.7983277Z 
2022-09-27T16:37:12.7983414Z Generating XML reports...
2022-09-27T16:37:12.8023198Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163705.xml
2022-09-27T16:37:13.5114957Z Running distributed/test_store ... [2022-09-27 16:37:13.510931]
2022-09-27T16:37:13.5115785Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:37:13.511035]
2022-09-27T16:37:15.4123066Z test_compare_set (__main__.FileStoreTest)
2022-09-27T16:37:15.4123548Z test_set_get (__main__.FileStoreTest)
2022-09-27T16:37:15.4123868Z test_compare_set (__main__.HashStoreTest)
2022-09-27T16:37:15.4124156Z test_set_get (__main__.HashStoreTest)
2022-09-27T16:37:15.4124504Z test_compare_set (__main__.PrefixFileStoreTest)
2022-09-27T16:37:15.4124869Z test_set_get (__main__.PrefixFileStoreTest)
2022-09-27T16:37:15.4125208Z test_get_underlying_store (__main__.PrefixStoreTest)
2022-09-27T16:37:15.4125575Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-09-27T16:37:15.4125911Z test_set_get (__main__.PrefixTCPStoreTest)
2022-09-27T16:37:15.4126229Z test_set_get (__main__.PythonStoreTest)
2022-09-27T16:37:15.4126549Z test_nominal (__main__.RendezvousEnvTest)
2022-09-27T16:37:15.4127705Z test_common_errors (__main__.RendezvousFileTest)
2022-09-27T16:37:15.4129099Z test_nominal (__main__.RendezvousFileTest)
2022-09-27T16:37:15.4129466Z test_common_errors (__main__.RendezvousTCPTest)
2022-09-27T16:37:15.4130124Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-09-27T16:37:15.4130736Z test_nominal (__main__.RendezvousTCPTest)
2022-09-27T16:37:15.4131097Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-09-27T16:37:15.4131472Z test_unknown_handler (__main__.RendezvousTest)
2022-09-27T16:37:15.4131799Z test_url_with_node_params (__main__.RendezvousTest)
2022-09-27T16:37:15.4132161Z test_address_already_in_use (__main__.TCPStoreTest)
2022-09-27T16:37:15.4179864Z test_compare_set (__main__.TCPStoreTest)
2022-09-27T16:37:15.4180385Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-09-27T16:37:15.4180756Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-09-27T16:37:15.4181155Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-09-27T16:37:15.4181522Z test_multitenancy (__main__.TCPStoreTest)
2022-09-27T16:37:15.4181839Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-09-27T16:37:15.4182161Z test_set_get (__main__.TCPStoreTest)
2022-09-27T16:37:17.0235902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:17.0236471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:17.0239009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:17.0239538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:17.2763453Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:17.2779096Z 
2022-09-27T16:37:17.2779343Z Running tests...
2022-09-27T16:37:17.2779800Z ----------------------------------------------------------------------
2022-09-27T16:37:18.8986444Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:18.9792491Z ok (1.701s)
2022-09-27T16:37:18.9792943Z 
2022-09-27T16:37:18.9793461Z ----------------------------------------------------------------------
2022-09-27T16:37:18.9793856Z Ran 1 test in 1.701s
2022-09-27T16:37:18.9794036Z 
2022-09-27T16:37:18.9794135Z OK
2022-09-27T16:37:18.9794283Z 
2022-09-27T16:37:18.9794397Z Generating XML reports...
2022-09-27T16:37:18.9832018Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927163717.xml
2022-09-27T16:37:21.0268984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:21.0269516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:21.0270598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:21.0271108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:21.2810005Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:21.2825523Z 
2022-09-27T16:37:21.2825998Z Running tests...
2022-09-27T16:37:21.2826460Z ----------------------------------------------------------------------
2022-09-27T16:37:22.8813074Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:22.9008146Z ok (1.618s)
2022-09-27T16:37:22.9010577Z 
2022-09-27T16:37:22.9011482Z ----------------------------------------------------------------------
2022-09-27T16:37:22.9011868Z Ran 1 test in 1.618s
2022-09-27T16:37:22.9012048Z 
2022-09-27T16:37:22.9012151Z OK
2022-09-27T16:37:22.9012295Z 
2022-09-27T16:37:22.9012437Z Generating XML reports...
2022-09-27T16:37:22.9692430Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927163721.xml
2022-09-27T16:37:25.0147343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:25.0149215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:25.0149879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:25.0150716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:25.2696744Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:25.2711225Z 
2022-09-27T16:37:25.2711459Z Running tests...
2022-09-27T16:37:25.2712203Z ----------------------------------------------------------------------
2022-09-27T16:37:26.8655694Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:26.8852528Z ok (1.614s)
2022-09-27T16:37:26.8854360Z 
2022-09-27T16:37:26.8854928Z ----------------------------------------------------------------------
2022-09-27T16:37:26.8855280Z Ran 1 test in 1.614s
2022-09-27T16:37:26.8855455Z 
2022-09-27T16:37:26.8855560Z OK
2022-09-27T16:37:26.8855728Z 
2022-09-27T16:37:26.8855863Z Generating XML reports...
2022-09-27T16:37:26.9536838Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927163725.xml
2022-09-27T16:37:29.0326351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:29.0326910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:29.0328347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:29.0328843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:29.2798090Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:29.2812549Z 
2022-09-27T16:37:29.2812885Z Running tests...
2022-09-27T16:37:29.2813390Z ----------------------------------------------------------------------
2022-09-27T16:37:30.8367971Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:30.8546477Z ok (1.573s)
2022-09-27T16:37:30.8547793Z 
2022-09-27T16:37:30.8548577Z ----------------------------------------------------------------------
2022-09-27T16:37:30.8548954Z Ran 1 test in 1.574s
2022-09-27T16:37:30.8549101Z 
2022-09-27T16:37:30.8549205Z OK
2022-09-27T16:37:30.8549347Z 
2022-09-27T16:37:30.8549489Z Generating XML reports...
2022-09-27T16:37:30.9207643Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927163729.xml
2022-09-27T16:37:32.9183356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:32.9183856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:32.9184676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:32.9185207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:33.1571222Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:33.1585806Z 
2022-09-27T16:37:33.1586129Z Running tests...
2022-09-27T16:37:33.1586579Z ----------------------------------------------------------------------
2022-09-27T16:37:34.7196884Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:34.8003071Z ok (1.641s)
2022-09-27T16:37:34.8003866Z 
2022-09-27T16:37:34.8004308Z ----------------------------------------------------------------------
2022-09-27T16:37:34.8004701Z Ran 1 test in 1.642s
2022-09-27T16:37:34.8004877Z 
2022-09-27T16:37:34.8004979Z OK
2022-09-27T16:37:34.8005128Z 
2022-09-27T16:37:34.8005246Z Generating XML reports...
2022-09-27T16:37:34.8042054Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927163733.xml
2022-09-27T16:37:36.8459218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:36.8459929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:36.8460919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:36.8461403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:37.0880212Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:37.0894333Z 
2022-09-27T16:37:37.0894514Z Running tests...
2022-09-27T16:37:37.0894969Z ----------------------------------------------------------------------
2022-09-27T16:37:38.6790274Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:38.6979786Z ok (1.608s)
2022-09-27T16:37:38.6982142Z 
2022-09-27T16:37:38.6982862Z ----------------------------------------------------------------------
2022-09-27T16:37:38.6983279Z Ran 1 test in 1.609s
2022-09-27T16:37:38.6983432Z 
2022-09-27T16:37:38.6983524Z OK
2022-09-27T16:37:38.6983674Z 
2022-09-27T16:37:38.6983833Z Generating XML reports...
2022-09-27T16:37:38.7643405Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927163737.xml
2022-09-27T16:37:40.8125733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:40.8126222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:40.8127276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:40.8127805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:41.0697344Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:41.0712863Z 
2022-09-27T16:37:41.0713011Z Running tests...
2022-09-27T16:37:41.0713705Z ----------------------------------------------------------------------
2022-09-27T16:37:41.0748220Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2022-09-27T16:37:41.0749013Z 
2022-09-27T16:37:41.0749322Z ----------------------------------------------------------------------
2022-09-27T16:37:41.0749687Z Ran 1 test in 0.004s
2022-09-27T16:37:41.0749886Z 
2022-09-27T16:37:41.0749980Z OK
2022-09-27T16:37:41.0750104Z 
2022-09-27T16:37:41.0750240Z Generating XML reports...
2022-09-27T16:37:41.0784267Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20220927163741.xml
2022-09-27T16:37:43.0040866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:43.0041901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:43.0043128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:43.0044087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:43.2497520Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:43.2511099Z 
2022-09-27T16:37:43.2511353Z Running tests...
2022-09-27T16:37:43.2511807Z ----------------------------------------------------------------------
2022-09-27T16:37:44.8089948Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:44.8916522Z ok (1.640s)
2022-09-27T16:37:44.8920278Z 
2022-09-27T16:37:44.8920680Z ----------------------------------------------------------------------
2022-09-27T16:37:44.8921083Z Ran 1 test in 1.641s
2022-09-27T16:37:44.8921258Z 
2022-09-27T16:37:44.8921361Z OK
2022-09-27T16:37:44.8921503Z 
2022-09-27T16:37:44.8921646Z Generating XML reports...
2022-09-27T16:37:44.8960266Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927163743.xml
2022-09-27T16:37:46.9080443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:46.9081408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:46.9082413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:46.9082856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:47.1636280Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:47.1652664Z 
2022-09-27T16:37:47.1653088Z Running tests...
2022-09-27T16:37:47.1653631Z ----------------------------------------------------------------------
2022-09-27T16:37:48.7522134Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:48.7724477Z ok (1.607s)
2022-09-27T16:37:48.7727528Z 
2022-09-27T16:37:48.7728032Z ----------------------------------------------------------------------
2022-09-27T16:37:48.7728409Z Ran 1 test in 1.607s
2022-09-27T16:37:48.7728584Z 
2022-09-27T16:37:48.7728674Z OK
2022-09-27T16:37:48.7728815Z 
2022-09-27T16:37:48.7728959Z Generating XML reports...
2022-09-27T16:37:48.8423607Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927163747.xml
2022-09-27T16:37:50.8483078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:50.8483775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:50.8485040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:50.8485559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:51.0994213Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:51.1009320Z 
2022-09-27T16:37:51.1009490Z Running tests...
2022-09-27T16:37:51.1010202Z ----------------------------------------------------------------------
2022-09-27T16:37:52.7090020Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:52.7261908Z ok (1.625s)
2022-09-27T16:37:52.7262763Z 
2022-09-27T16:37:52.7263367Z ----------------------------------------------------------------------
2022-09-27T16:37:52.7264087Z Ran 1 test in 1.625s
2022-09-27T16:37:52.7264374Z 
2022-09-27T16:37:52.7264518Z OK
2022-09-27T16:37:52.7264666Z 
2022-09-27T16:37:52.7264801Z Generating XML reports...
2022-09-27T16:37:52.7913075Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220927163751.xml
2022-09-27T16:37:54.8304021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:54.8304533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:54.8305358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:54.8305842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:55.0745110Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:55.0758795Z 
2022-09-27T16:37:55.0759274Z Running tests...
2022-09-27T16:37:55.0759801Z ----------------------------------------------------------------------
2022-09-27T16:37:56.6617132Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:56.6808323Z ok (1.605s)
2022-09-27T16:37:56.6809111Z 
2022-09-27T16:37:56.6809538Z ----------------------------------------------------------------------
2022-09-27T16:37:56.6810150Z Ran 1 test in 1.605s
2022-09-27T16:37:56.6810336Z 
2022-09-27T16:37:56.6810418Z OK
2022-09-27T16:37:56.6810561Z 
2022-09-27T16:37:56.6810703Z Generating XML reports...
2022-09-27T16:37:56.7468447Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220927163755.xml
2022-09-27T16:37:58.7745072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:58.7745890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:58.7747284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:58.7747806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:59.0235922Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:37:59.0250612Z 
2022-09-27T16:37:59.0250821Z Running tests...
2022-09-27T16:37:59.0251510Z ----------------------------------------------------------------------
2022-09-27T16:38:00.6336452Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:00.6514274Z ok (1.626s)
2022-09-27T16:38:00.6515180Z 
2022-09-27T16:38:00.6515975Z ----------------------------------------------------------------------
2022-09-27T16:38:00.6516566Z Ran 1 test in 1.626s
2022-09-27T16:38:00.6516789Z 
2022-09-27T16:38:00.6516892Z OK
2022-09-27T16:38:00.6517044Z 
2022-09-27T16:38:00.6519699Z Generating XML reports...
2022-09-27T16:38:00.7227249Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927163759.xml
2022-09-27T16:38:02.8264170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:02.8265132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:02.8266283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:02.8267185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:03.0823486Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:03.0837264Z 
2022-09-27T16:38:03.0837634Z Running tests...
2022-09-27T16:38:03.0838167Z ----------------------------------------------------------------------
2022-09-27T16:38:04.6815386Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:04.6998587Z ok (1.616s)
2022-09-27T16:38:04.6999155Z 
2022-09-27T16:38:04.6999553Z ----------------------------------------------------------------------
2022-09-27T16:38:04.6999914Z Ran 1 test in 1.616s
2022-09-27T16:38:04.7000088Z 
2022-09-27T16:38:04.7000190Z OK
2022-09-27T16:38:04.7000334Z 
2022-09-27T16:38:04.7000482Z Generating XML reports...
2022-09-27T16:38:04.7639309Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927163803.xml
2022-09-27T16:38:06.7952683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:06.7953216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:06.7953816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:06.7954332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:07.0424278Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:07.0438644Z 
2022-09-27T16:38:07.0438933Z Running tests...
2022-09-27T16:38:07.0439391Z ----------------------------------------------------------------------
2022-09-27T16:38:08.6182780Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:08.6353978Z ok (1.591s)
2022-09-27T16:38:08.6355361Z 
2022-09-27T16:38:08.6356171Z ----------------------------------------------------------------------
2022-09-27T16:38:08.6356548Z Ran 1 test in 1.592s
2022-09-27T16:38:08.6356728Z 
2022-09-27T16:38:08.6356856Z OK
2022-09-27T16:38:08.6357007Z 
2022-09-27T16:38:08.6357120Z Generating XML reports...
2022-09-27T16:38:08.7012221Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163807.xml
2022-09-27T16:38:10.7640511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:10.7641001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:10.7644509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:10.7645052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:11.0105326Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:11.0119171Z 
2022-09-27T16:38:11.0119388Z Running tests...
2022-09-27T16:38:11.0119858Z ----------------------------------------------------------------------
2022-09-27T16:38:12.6032058Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:12.6524827Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-09-27T16:38:12.6525396Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-09-27T16:38:12.6528540Z ok (1.641s)
2022-09-27T16:38:12.6530130Z 
2022-09-27T16:38:12.6530630Z ----------------------------------------------------------------------
2022-09-27T16:38:12.6530973Z Ran 1 test in 1.641s
2022-09-27T16:38:12.6531150Z 
2022-09-27T16:38:12.6531260Z OK
2022-09-27T16:38:12.6531372Z 
2022-09-27T16:38:12.6531507Z Generating XML reports...
2022-09-27T16:38:12.7184506Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163811.xml
2022-09-27T16:38:14.7604110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:14.7604952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:14.7606027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:14.7606534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:15.0122137Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:15.0137578Z 
2022-09-27T16:38:15.0137956Z Running tests...
2022-09-27T16:38:15.0138467Z ----------------------------------------------------------------------
2022-09-27T16:38:16.6239744Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:16.6436927Z ok (1.630s)
2022-09-27T16:38:16.6438855Z 
2022-09-27T16:38:16.6439619Z ----------------------------------------------------------------------
2022-09-27T16:38:16.6440298Z Ran 1 test in 1.630s
2022-09-27T16:38:16.6440485Z 
2022-09-27T16:38:16.6440598Z OK
2022-09-27T16:38:16.6440747Z 
2022-09-27T16:38:16.6440860Z Generating XML reports...
2022-09-27T16:38:16.7138987Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163815.xml
2022-09-27T16:38:18.7861317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.7861867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.7864167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.7864672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:19.0370043Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:19.0385411Z 
2022-09-27T16:38:19.0385666Z Running tests...
2022-09-27T16:38:19.0386124Z ----------------------------------------------------------------------
2022-09-27T16:38:20.6467547Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:30.7496950Z ok (11.711s)
2022-09-27T16:38:30.7497383Z 
2022-09-27T16:38:30.7498169Z ----------------------------------------------------------------------
2022-09-27T16:38:30.7498623Z Ran 1 test in 11.711s
2022-09-27T16:38:30.7498800Z 
2022-09-27T16:38:30.7501074Z OK
2022-09-27T16:38:30.7501446Z 
2022-09-27T16:38:30.7501747Z Generating XML reports...
2022-09-27T16:38:30.8236349Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163819.xml
2022-09-27T16:38:32.8980494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:32.8981104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:32.8982456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:32.8983181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:33.1472929Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:33.1486836Z 
2022-09-27T16:38:33.1487230Z Running tests...
2022-09-27T16:38:33.1488187Z ----------------------------------------------------------------------
2022-09-27T16:38:34.7464181Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:34.7637709Z ok (1.615s)
2022-09-27T16:38:34.7638898Z 
2022-09-27T16:38:34.7639649Z ----------------------------------------------------------------------
2022-09-27T16:38:34.7640342Z Ran 1 test in 1.615s
2022-09-27T16:38:34.7640702Z 
2022-09-27T16:38:34.7640899Z OK
2022-09-27T16:38:34.7641138Z 
2022-09-27T16:38:34.7641332Z Generating XML reports...
2022-09-27T16:38:34.8323571Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927163833.xml
2022-09-27T16:38:36.8899912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:36.8900479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:36.8902137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:36.8903007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:37.1476225Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:37.1490174Z 
2022-09-27T16:38:37.1490505Z Running tests...
2022-09-27T16:38:37.1491449Z ----------------------------------------------------------------------
2022-09-27T16:38:38.7636297Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:38.7812876Z ok (1.632s)
2022-09-27T16:38:38.7814187Z 
2022-09-27T16:38:38.7814872Z ----------------------------------------------------------------------
2022-09-27T16:38:38.7815651Z Ran 1 test in 1.632s
2022-09-27T16:38:38.7815977Z 
2022-09-27T16:38:38.7816101Z OK
2022-09-27T16:38:38.7816221Z 
2022-09-27T16:38:38.7816357Z Generating XML reports...
2022-09-27T16:38:38.8528952Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927163837.xml
2022-09-27T16:38:40.9423858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:40.9424430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:40.9425771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:40.9426263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:41.1893439Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:41.1907552Z 
2022-09-27T16:38:41.1907702Z Running tests...
2022-09-27T16:38:41.1908159Z ----------------------------------------------------------------------
2022-09-27T16:38:42.7554706Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:42.7731823Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:44537 (errno: 98 - Address already in use).
2022-09-27T16:38:42.7750266Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:44537 (errno: 98 - Address already in use).
2022-09-27T16:38:42.7750734Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-09-27T16:38:42.7755786Z ok (1.585s)
2022-09-27T16:38:42.7757488Z 
2022-09-27T16:38:42.7757926Z ----------------------------------------------------------------------
2022-09-27T16:38:42.7758296Z Ran 1 test in 1.585s
2022-09-27T16:38:42.7758462Z 
2022-09-27T16:38:42.7758559Z OK
2022-09-27T16:38:42.7758703Z 
2022-09-27T16:38:42.7758838Z Generating XML reports...
2022-09-27T16:38:42.8472857Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163841.xml
2022-09-27T16:38:44.8460834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:44.8461395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:44.8462753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:44.8463271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:45.0953777Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:45.0967302Z 
2022-09-27T16:38:45.0967858Z Running tests...
2022-09-27T16:38:45.0968399Z ----------------------------------------------------------------------
2022-09-27T16:38:46.6602300Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:46.7437972Z ok (1.647s)
2022-09-27T16:38:46.7438458Z 
2022-09-27T16:38:46.7439189Z ----------------------------------------------------------------------
2022-09-27T16:38:46.7439544Z Ran 1 test in 1.647s
2022-09-27T16:38:46.7439694Z 
2022-09-27T16:38:46.7439797Z OK
2022-09-27T16:38:46.7439942Z 
2022-09-27T16:38:46.7440081Z Generating XML reports...
2022-09-27T16:38:46.7476402Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163845.xml
2022-09-27T16:38:48.7730066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:48.7730617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:48.7732988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:48.7733529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:49.0200016Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:49.0213873Z 
2022-09-27T16:38:49.0214298Z Running tests...
2022-09-27T16:38:49.0214901Z ----------------------------------------------------------------------
2022-09-27T16:38:50.5910184Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:50.6101310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:50.6102156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:38:50.6844302Z ok (1.663s)
2022-09-27T16:38:50.6844881Z 
2022-09-27T16:38:50.6845445Z ----------------------------------------------------------------------
2022-09-27T16:38:50.6845825Z Ran 1 test in 1.663s
2022-09-27T16:38:50.6846138Z 
2022-09-27T16:38:50.6846374Z OK
2022-09-27T16:38:50.6846667Z 
2022-09-27T16:38:50.6846936Z Generating XML reports...
2022-09-27T16:38:50.7558586Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163849.xml
2022-09-27T16:38:52.8510171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:52.8510692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:52.8511987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:52.8512526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:53.1124279Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:53.1139623Z 
2022-09-27T16:38:53.1140276Z Running tests...
2022-09-27T16:38:53.1140878Z ----------------------------------------------------------------------
2022-09-27T16:38:54.7430909Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:54.7678851Z ok (1.654s)
2022-09-27T16:38:54.7680658Z 
2022-09-27T16:38:54.7681242Z ----------------------------------------------------------------------
2022-09-27T16:38:54.7681630Z Ran 1 test in 1.654s
2022-09-27T16:38:54.7681824Z 
2022-09-27T16:38:54.7681898Z OK
2022-09-27T16:38:54.7682002Z 
2022-09-27T16:38:54.7682144Z Generating XML reports...
2022-09-27T16:38:54.8413623Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163853.xml
2022-09-27T16:38:56.9137721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:56.9138833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:56.9140445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:56.9141477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:57.1559650Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:38:57.1573357Z 
2022-09-27T16:38:57.1573582Z Running tests...
2022-09-27T16:38:57.1574011Z ----------------------------------------------------------------------
2022-09-27T16:38:58.7211333Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:58.7403904Z ok (1.583s)
2022-09-27T16:38:58.7405484Z 
2022-09-27T16:38:58.7405900Z ----------------------------------------------------------------------
2022-09-27T16:38:58.7406279Z Ran 1 test in 1.583s
2022-09-27T16:38:58.7406879Z 
2022-09-27T16:38:58.7406985Z OK
2022-09-27T16:38:58.7407133Z 
2022-09-27T16:38:58.7407275Z Generating XML reports...
2022-09-27T16:38:58.8059543Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163857.xml
2022-09-27T16:39:00.8625132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:00.8626197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:00.8627402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:00.8628332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:01.1059306Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:39:01.1072899Z 
2022-09-27T16:39:01.1073356Z Running tests...
2022-09-27T16:39:01.1073857Z ----------------------------------------------------------------------
2022-09-27T16:39:02.6783673Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:02.6969717Z ok (1.590s)
2022-09-27T16:39:02.6971008Z 
2022-09-27T16:39:02.6971475Z ----------------------------------------------------------------------
2022-09-27T16:39:02.6971830Z Ran 1 test in 1.590s
2022-09-27T16:39:02.6972007Z 
2022-09-27T16:39:02.6972422Z OK
2022-09-27T16:39:02.6972570Z 
2022-09-27T16:39:02.6972703Z Generating XML reports...
2022-09-27T16:39:02.7747857Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163901.xml
2022-09-27T16:39:04.8064034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:04.8064575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:04.8065567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:04.8066035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:05.0523339Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:39:05.0537455Z 
2022-09-27T16:39:05.0537759Z Running tests...
2022-09-27T16:39:05.0538223Z ----------------------------------------------------------------------
2022-09-27T16:39:06.6072985Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:08.6387839Z ok (3.585s)
2022-09-27T16:39:08.6388148Z 
2022-09-27T16:39:08.6388589Z ----------------------------------------------------------------------
2022-09-27T16:39:08.6388876Z Ran 1 test in 3.585s
2022-09-27T16:39:08.6389010Z 
2022-09-27T16:39:08.6389111Z OK
2022-09-27T16:39:08.6389255Z 
2022-09-27T16:39:08.6389437Z Generating XML reports...
2022-09-27T16:39:08.7078270Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163905.xml
2022-09-27T16:39:10.7733451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:10.7733981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:10.7735589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:10.7736103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:11.0186499Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-09-27T16:39:11.0200662Z 
2022-09-27T16:39:11.0201016Z Running tests...
2022-09-27T16:39:11.0201509Z ----------------------------------------------------------------------
2022-09-27T16:39:12.5901323Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:12.6107336Z ok (1.590s)
2022-09-27T16:39:12.6108502Z 
2022-09-27T16:39:12.6109035Z ----------------------------------------------------------------------
2022-09-27T16:39:12.6109381Z Ran 1 test in 1.591s
2022-09-27T16:39:12.6109560Z 
2022-09-27T16:39:12.6109659Z OK
2022-09-27T16:39:12.6109761Z 
2022-09-27T16:39:12.6109892Z Generating XML reports...
2022-09-27T16:39:12.6761524Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163911.xml
2022-09-27T16:39:13.3795853Z Running distributed/algorithms/test_join ... [2022-09-27 16:39:13.379044]
2022-09-27T16:39:13.3796675Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:39:13.379144]
2022-09-27T16:39:15.2654788Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-09-27T16:39:15.2672830Z 
2022-09-27T16:39:15.2673097Z Running tests...
2022-09-27T16:39:15.2673567Z ----------------------------------------------------------------------
2022-09-27T16:39:15.2680222Z   test_join_kwargs (__main__.TestJoin)
2022-09-27T16:39:16.8338147Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:16.9144773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53269
2022-09-27T16:39:16.9150309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53270
2022-09-27T16:39:18.5869432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:18.5870410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:18.5871978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:18.5872502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:18.6155613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:18.6156115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:18.6160031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:18.6160539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:18.8747148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:18.8750319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:18.8764223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:18.8768392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:18.8769760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:18.8853755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:22.0271644Z ok (6.760s)
2022-09-27T16:39:22.0277539Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-09-27T16:39:22.0293713Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53351
2022-09-27T16:39:22.0300276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53352
2022-09-27T16:39:23.6607874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:23.6608370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:23.6609319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:23.6610067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:23.6821071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:23.6821528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:23.6824985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:23.6825477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:23.9401730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:23.9404361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:23.9441190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:23.9445284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:23.9446395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:23.9507555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:27.0410546Z ok (5.014s)
2022-09-27T16:39:27.0417114Z   test_multiple_joinables (__main__.TestJoin)
2022-09-27T16:39:27.0433357Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53433
2022-09-27T16:39:27.0439791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53434
2022-09-27T16:39:28.7023905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:28.7024886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:28.7025999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:28.7026971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:28.7144586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:28.7145060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:28.7148743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:28.7149261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:28.9767124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:28.9768968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:28.9786474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:28.9791184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:28.9792332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:28.9872388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:32.1549207Z ok (5.114s)
2022-09-27T16:39:32.1554073Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-09-27T16:39:32.1570659Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53515
2022-09-27T16:39:32.1577968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53516
2022-09-27T16:39:33.8279857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:33.8280895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:33.8282053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:33.8282990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:33.8372499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:33.8372998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:33.8377057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:33.8377536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:34.1024315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:34.1025415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:34.1026653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:34.1029071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:34.1029940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:34.1130940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:37.2690600Z ok (5.114s)
2022-09-27T16:39:37.2697302Z   test_single_joinable (__main__.TestJoin)
2022-09-27T16:39:37.2713867Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53597
2022-09-27T16:39:37.2722213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53598
2022-09-27T16:39:38.9352357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:38.9353393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:38.9354649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:38.9355596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:38.9520799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:38.9521759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:38.9526630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:38.9527652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:39.2260657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:39.2263321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:39.2301229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:39.2307079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:39.2308464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:39.2368918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:42.4841618Z ok (5.215s)
2022-09-27T16:39:42.4846823Z   test_single_joinable_disable (__main__.TestJoin)
2022-09-27T16:39:42.4864423Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53679
2022-09-27T16:39:42.4871406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53680
2022-09-27T16:39:44.1590767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:44.1591294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:44.1592624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:44.1593128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:44.1716988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:44.1717440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:44.1721385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:44.1721892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:44.4436573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:44.4437163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:44.4438552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:44.4440796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:44.4442072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:44.4542140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:47.6990005Z ok (5.215s)
2022-09-27T16:39:47.6996437Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-09-27T16:39:47.7013463Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53761
2022-09-27T16:39:47.7299438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53762
2022-09-27T16:39:49.4221653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:49.4222201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:49.4223783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:49.4224276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:49.4551744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:49.4552219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:49.4556370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:49.4556925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:49.7085942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:49.7088798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:49.7251463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:49.7256111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:49.7257318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:49.7293550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:52.9423215Z ok (5.243s)
2022-09-27T16:39:52.9427885Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-09-27T16:39:52.9444980Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53843
2022-09-27T16:39:52.9451679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53844
2022-09-27T16:39:54.6322410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:54.6322969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:54.6323998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:54.6324495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:54.6420939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:54.6421382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:54.6425620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:54.6426056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:54.9150105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:54.9152420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:54.9166699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:54.9171322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:54.9172189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:54.9255483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:58.1571107Z ok (5.215s)
2022-09-27T16:39:58.1576284Z   test_single_joinable_throw (__main__.TestJoin)
2022-09-27T16:39:58.1593230Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53925
2022-09-27T16:39:58.1600586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53926
2022-09-27T16:39:59.8844522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:59.8845479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:59.8847165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:59.8848096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:59.8868275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:59.8869316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:59.8874057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:59.8875095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:00.1466222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:00.1467830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:00.1685830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:00.1690619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:00.1691790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:00.1775346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:03.4719558Z ok (5.315s)
2022-09-27T16:40:03.4719823Z 
2022-09-27T16:40:03.4720296Z ----------------------------------------------------------------------
2022-09-27T16:40:03.4723728Z Ran 9 tests in 48.204s
2022-09-27T16:40:03.4723948Z 
2022-09-27T16:40:03.4724236Z OK
2022-09-27T16:40:03.4724385Z 
2022-09-27T16:40:03.4724533Z Generating XML reports...
2022-09-27T16:40:03.4769961Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220927163915.xml
2022-09-27T16:40:03.9271825Z Running distributed/test_c10d_spawn_gloo ... [2022-09-27 16:40:03.926594]
2022-09-27T16:40:03.9272612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:40:03.926697]
2022-09-27T16:40:05.8492821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2aytapvy
2022-09-27T16:40:05.8493724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2aytapvy/_remote_module_non_scriptable.py
2022-09-27T16:40:07.4466398Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:07.4518787Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-09-27T16:40:07.4519935Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:40:07.4520416Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:40:07.4520856Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-09-27T16:40:07.4521247Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:40:07.4521580Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:40:07.4522723Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-09-27T16:40:07.4523916Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4524306Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4524732Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4525147Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4525529Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4525936Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4526328Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:07.4526724Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-09-27T16:40:09.0855881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:09.0856561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:09.0857554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:09.0858070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:09.3278601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7tdz1or
2022-09-27T16:40:09.3280153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7tdz1or/_remote_module_non_scriptable.py
2022-09-27T16:40:10.9231182Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:10.9310244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:10.9325140Z 
2022-09-27T16:40:10.9325381Z Running tests...
2022-09-27T16:40:10.9326095Z ----------------------------------------------------------------------
2022-09-27T16:40:10.9402226Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:10.9556030Z ok (0.023s)
2022-09-27T16:40:10.9558128Z 
2022-09-27T16:40:10.9558816Z ----------------------------------------------------------------------
2022-09-27T16:40:10.9559191Z Ran 1 test in 0.023s
2022-09-27T16:40:10.9559342Z 
2022-09-27T16:40:10.9559452Z OK
2022-09-27T16:40:10.9559596Z 
2022-09-27T16:40:10.9559731Z Generating XML reports...
2022-09-27T16:40:10.9592463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164010.xml
2022-09-27T16:40:13.0523368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:13.0523881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:13.0525102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:13.0525895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:13.2934466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjz2ui0m
2022-09-27T16:40:13.2935947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjz2ui0m/_remote_module_non_scriptable.py
2022-09-27T16:40:14.8978895Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:14.9052201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:14.9068149Z 
2022-09-27T16:40:14.9068415Z Running tests...
2022-09-27T16:40:14.9068854Z ----------------------------------------------------------------------
2022-09-27T16:40:15.3526099Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:15.3738606Z ok (0.467s)
2022-09-27T16:40:15.3741196Z 
2022-09-27T16:40:15.3741995Z ----------------------------------------------------------------------
2022-09-27T16:40:15.3742803Z Ran 1 test in 0.467s
2022-09-27T16:40:15.3743041Z 
2022-09-27T16:40:15.3743213Z OK
2022-09-27T16:40:15.3743401Z 
2022-09-27T16:40:15.3743543Z Generating XML reports...
2022-09-27T16:40:15.3781829Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164014.xml
2022-09-27T16:40:17.4801922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:17.4802577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:17.4803438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:17.4804226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:17.7287494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnlnftbo
2022-09-27T16:40:17.7288134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnlnftbo/_remote_module_non_scriptable.py
2022-09-27T16:40:19.3730567Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:19.3804917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:19.3821072Z 
2022-09-27T16:40:19.3821488Z Running tests...
2022-09-27T16:40:19.3822000Z ----------------------------------------------------------------------
2022-09-27T16:40:20.2228383Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:20.6997398Z ok (1.317s)
2022-09-27T16:40:20.6998052Z 
2022-09-27T16:40:20.6998662Z ----------------------------------------------------------------------
2022-09-27T16:40:20.6999014Z Ran 1 test in 1.318s
2022-09-27T16:40:20.6999203Z 
2022-09-27T16:40:20.6999278Z OK
2022-09-27T16:40:20.6999389Z 
2022-09-27T16:40:20.6999522Z Generating XML reports...
2022-09-27T16:40:20.7039577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164019.xml
2022-09-27T16:40:22.9439803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:22.9440885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:22.9442092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:22.9443023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:23.1847915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ebnxgk5
2022-09-27T16:40:23.1848859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ebnxgk5/_remote_module_non_scriptable.py
2022-09-27T16:40:24.7775767Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:24.7850991Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:24.7867220Z 
2022-09-27T16:40:24.7867718Z Running tests...
2022-09-27T16:40:24.7868226Z ----------------------------------------------------------------------
2022-09-27T16:40:24.8653522Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54193
2022-09-27T16:40:24.8659334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54194
2022-09-27T16:40:26.5348144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:26.5348693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:26.5349942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:26.5350413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:26.5365551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:26.5366039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:26.5370580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:26.5371058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:26.7804885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivjq07v_
2022-09-27T16:40:26.7806159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivjq07v_/_remote_module_non_scriptable.py
2022-09-27T16:40:26.7942096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxhq2fes
2022-09-27T16:40:26.7944634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxhq2fes/_remote_module_non_scriptable.py
2022-09-27T16:40:28.4853761Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:28.4891730Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:28.4906992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:28.4942157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:28.5053586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:28.5054054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:28.5054944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:28.5055612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:30.3784915Z ok (5.591s)
2022-09-27T16:40:30.3785223Z 
2022-09-27T16:40:30.3785666Z ----------------------------------------------------------------------
2022-09-27T16:40:30.3786090Z Ran 1 test in 5.592s
2022-09-27T16:40:30.3786208Z 
2022-09-27T16:40:30.3786304Z OK
2022-09-27T16:40:30.3786480Z 
2022-09-27T16:40:30.3786625Z Generating XML reports...
2022-09-27T16:40:30.3826537Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164024.xml
2022-09-27T16:40:32.4935896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:32.4936776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:32.4938093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:32.4938613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:32.7357416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66vsmh5m
2022-09-27T16:40:32.7358859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66vsmh5m/_remote_module_non_scriptable.py
2022-09-27T16:40:34.3386900Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:34.3461839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:34.3478127Z 
2022-09-27T16:40:34.3478592Z Running tests...
2022-09-27T16:40:34.3479105Z ----------------------------------------------------------------------
2022-09-27T16:40:34.4313742Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54309
2022-09-27T16:40:34.4318691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54310
2022-09-27T16:40:36.0167211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:36.0167746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:36.0168986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:36.0169477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:36.1007707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:36.1008162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:36.1011789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:36.1012265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:36.2637127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0i1jl6p
2022-09-27T16:40:36.2638037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0i1jl6p/_remote_module_non_scriptable.py
2022-09-27T16:40:36.3847486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvmzk53p
2022-09-27T16:40:36.3848442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvmzk53p/_remote_module_non_scriptable.py
2022-09-27T16:40:37.9477192Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:37.9530199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:38.0306807Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:38.0358162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:38.0549302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:38.0550086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:38.0550975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:38.0551703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:39.9472114Z ok (5.599s)
2022-09-27T16:40:39.9472390Z 
2022-09-27T16:40:39.9472807Z ----------------------------------------------------------------------
2022-09-27T16:40:39.9473156Z Ran 1 test in 5.599s
2022-09-27T16:40:39.9473328Z 
2022-09-27T16:40:39.9476240Z OK
2022-09-27T16:40:39.9476406Z 
2022-09-27T16:40:39.9476929Z Generating XML reports...
2022-09-27T16:40:39.9512815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164034.xml
2022-09-27T16:40:42.0926051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:42.0926616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:42.0927521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:42.0928291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:42.3334955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13_4tt18
2022-09-27T16:40:42.3335751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13_4tt18/_remote_module_non_scriptable.py
2022-09-27T16:40:43.9304380Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:43.9376478Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:43.9392031Z 
2022-09-27T16:40:43.9392407Z Running tests...
2022-09-27T16:40:43.9392904Z ----------------------------------------------------------------------
2022-09-27T16:40:44.0413556Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54425
2022-09-27T16:40:44.0419186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54426
2022-09-27T16:40:45.7110964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:45.7111514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:45.7112410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:45.7112894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:45.7522777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:45.7523270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:45.7527746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:45.7528181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:45.9715948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5h4c1vs
2022-09-27T16:40:45.9716672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5h4c1vs/_remote_module_non_scriptable.py
2022-09-27T16:40:45.9992640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy16a9xj
2022-09-27T16:40:45.9995951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy16a9xj/_remote_module_non_scriptable.py
2022-09-27T16:40:47.6675140Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:47.6729030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:47.7155498Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:47.7209069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:47.7344299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:47.7344848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:47.7345670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:47.7346352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:49.5550257Z ok (5.615s)
2022-09-27T16:40:49.5550476Z 
2022-09-27T16:40:49.5550909Z ----------------------------------------------------------------------
2022-09-27T16:40:49.5551297Z Ran 1 test in 5.616s
2022-09-27T16:40:49.5551451Z 
2022-09-27T16:40:49.5551541Z OK
2022-09-27T16:40:49.5551683Z 
2022-09-27T16:40:49.5551823Z Generating XML reports...
2022-09-27T16:40:49.5589929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164043.xml
2022-09-27T16:40:51.6171506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:51.6172381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:51.6173448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:51.6173936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:51.8625043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1d50rjx
2022-09-27T16:40:51.8625650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1d50rjx/_remote_module_non_scriptable.py
2022-09-27T16:40:53.4574141Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:53.4648797Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:40:53.4668926Z 
2022-09-27T16:40:53.4669476Z Running tests...
2022-09-27T16:40:53.4669988Z ----------------------------------------------------------------------
2022-09-27T16:40:53.5504169Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54541
2022-09-27T16:40:53.5509203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54542
2022-09-27T16:40:55.1814192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:55.1814757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:55.1816795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:55.1817283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:55.2200713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:55.2201229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:55.2204626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:55.2205142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:55.4400373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaci0u2ok
2022-09-27T16:40:55.4401401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaci0u2ok/_remote_module_non_scriptable.py
2022-09-27T16:40:55.4665424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzhy8k2u
2022-09-27T16:40:55.4668285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzhy8k2u/_remote_module_non_scriptable.py
2022-09-27T16:40:57.1519111Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:57.1572769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:57.1921857Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:57.1973942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:57.2084367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:57.2084911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:57.2085682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:57.2086390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:59.0638616Z ok (5.596s)
2022-09-27T16:40:59.0638881Z 
2022-09-27T16:40:59.0639333Z ----------------------------------------------------------------------
2022-09-27T16:40:59.0639664Z Ran 1 test in 5.597s
2022-09-27T16:40:59.0639850Z 
2022-09-27T16:40:59.0639947Z OK
2022-09-27T16:40:59.0640082Z 
2022-09-27T16:40:59.0640229Z Generating XML reports...
2022-09-27T16:40:59.0680489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164053.xml
2022-09-27T16:41:01.2069664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.2070282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.2071705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.2072176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.4655910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm5o41lh
2022-09-27T16:41:01.4656572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm5o41lh/_remote_module_non_scriptable.py
2022-09-27T16:41:03.1025252Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:03.1097982Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:41:03.1113508Z 
2022-09-27T16:41:03.1113918Z Running tests...
2022-09-27T16:41:03.1114432Z ----------------------------------------------------------------------
2022-09-27T16:41:03.1927508Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54657
2022-09-27T16:41:03.1932800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54658
2022-09-27T16:41:04.8786973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:04.8787522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:04.8788828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:04.8789359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:04.9047836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:04.9048341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:04.9052839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:04.9053346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:05.1429851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskxuycyu
2022-09-27T16:41:05.1431110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskxuycyu/_remote_module_non_scriptable.py
2022-09-27T16:41:05.1542091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcryycpia
2022-09-27T16:41:05.1545305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcryycpia/_remote_module_non_scriptable.py
2022-09-27T16:41:06.8565651Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:06.8621587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:06.8773824Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:06.8825934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:06.8935054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:06.8935654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:06.8936464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:06.8937184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:08.7062563Z ok (5.594s)
2022-09-27T16:41:08.7062812Z 
2022-09-27T16:41:08.7063257Z ----------------------------------------------------------------------
2022-09-27T16:41:08.7063915Z Ran 1 test in 5.595s
2022-09-27T16:41:08.7064104Z 
2022-09-27T16:41:08.7064210Z OK
2022-09-27T16:41:08.7064340Z 
2022-09-27T16:41:08.7064496Z Generating XML reports...
2022-09-27T16:41:08.7101765Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164103.xml
2022-09-27T16:41:10.8232277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:10.8232884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:10.8233910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:10.8234398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:11.0767781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwwvz489
2022-09-27T16:41:11.0768926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwwvz489/_remote_module_non_scriptable.py
2022-09-27T16:41:12.6970446Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:12.7043061Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:41:12.7058855Z 
2022-09-27T16:41:12.7058985Z Running tests...
2022-09-27T16:41:12.7060080Z ----------------------------------------------------------------------
2022-09-27T16:41:12.7889264Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54773
2022-09-27T16:41:12.7895590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54774
2022-09-27T16:41:14.4635078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.4635624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.4636926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.4637467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:14.4935603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.4936065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.4940578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.4941079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:14.7235566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4n665uj
2022-09-27T16:41:14.7236429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4n665uj/_remote_module_non_scriptable.py
2022-09-27T16:41:14.7348082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xioh6v5
2022-09-27T16:41:14.7350984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xioh6v5/_remote_module_non_scriptable.py
2022-09-27T16:41:16.4082421Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:16.4135097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:16.4171338Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:16.4222763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:16.4434562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:16.4435093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:16.4435922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:16.4436632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:18.3026876Z ok (5.596s)
2022-09-27T16:41:18.3027329Z 
2022-09-27T16:41:18.3027814Z ----------------------------------------------------------------------
2022-09-27T16:41:18.3028161Z Ran 1 test in 5.597s
2022-09-27T16:41:18.3028407Z 
2022-09-27T16:41:18.3028588Z OK
2022-09-27T16:41:18.3028811Z 
2022-09-27T16:41:18.3028978Z Generating XML reports...
2022-09-27T16:41:18.3067922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164112.xml
2022-09-27T16:41:20.4204771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:20.4205663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:20.4206934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:20.4207447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:20.6631879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqttlmxp
2022-09-27T16:41:20.6632510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqttlmxp/_remote_module_non_scriptable.py
2022-09-27T16:41:22.2765880Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:22.2838553Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:41:22.2854884Z 
2022-09-27T16:41:22.2855302Z Running tests...
2022-09-27T16:41:22.2855819Z ----------------------------------------------------------------------
2022-09-27T16:41:22.3684434Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54889
2022-09-27T16:41:22.3688579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54890
2022-09-27T16:41:24.0742301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:24.0742928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:24.0743551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:24.0744054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:24.0783284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:24.0783765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:24.0787647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:24.0788167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:24.3281597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz1o3ama
2022-09-27T16:41:24.3282815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz1o3ama/_remote_module_non_scriptable.py
2022-09-27T16:41:24.3475977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjww1r9ay
2022-09-27T16:41:24.3479345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjww1r9ay/_remote_module_non_scriptable.py
2022-09-27T16:41:26.0700526Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:26.0748921Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:26.0757045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:26.0801585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:26.1013587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:26.1014184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:26.1015351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:26.1016075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:27.9821088Z ok (5.696s)
2022-09-27T16:41:27.9821353Z 
2022-09-27T16:41:27.9821764Z ----------------------------------------------------------------------
2022-09-27T16:41:27.9822177Z Ran 1 test in 5.696s
2022-09-27T16:41:27.9822356Z 
2022-09-27T16:41:27.9822456Z OK
2022-09-27T16:41:27.9822597Z 
2022-09-27T16:41:27.9822719Z Generating XML reports...
2022-09-27T16:41:27.9861382Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164122.xml
2022-09-27T16:41:30.0811100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:30.0811651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:30.0812906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:30.0813389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:30.3250755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0divcyxe
2022-09-27T16:41:30.3251907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0divcyxe/_remote_module_non_scriptable.py
2022-09-27T16:41:31.9354167Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:31.9428944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-09-27T16:41:31.9444505Z 
2022-09-27T16:41:31.9444819Z Running tests...
2022-09-27T16:41:31.9445586Z ----------------------------------------------------------------------
2022-09-27T16:41:32.0298002Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55005
2022-09-27T16:41:32.0302941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55006
2022-09-27T16:41:33.6949213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:33.6949735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:33.6951331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:33.6951824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:33.7465412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:33.7465895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:33.7469275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:33.7469994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:33.9463529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rkmqbj6
2022-09-27T16:41:33.9464421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rkmqbj6/_remote_module_non_scriptable.py
2022-09-27T16:41:33.9900177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vol9pjk
2022-09-27T16:41:33.9902538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vol9pjk/_remote_module_non_scriptable.py
2022-09-27T16:41:35.6041319Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:35.6094395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:35.6535368Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:35.6588507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:35.6699105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:35.6699631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:35.6700437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:35.6701131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:37.5433892Z ok (5.598s)
2022-09-27T16:41:37.5434289Z 
2022-09-27T16:41:37.5435014Z ----------------------------------------------------------------------
2022-09-27T16:41:37.5435577Z Ran 1 test in 5.599s
2022-09-27T16:41:37.5435856Z 
2022-09-27T16:41:37.5436006Z OK
2022-09-27T16:41:37.5436272Z 
2022-09-27T16:41:37.5436504Z Generating XML reports...
2022-09-27T16:41:37.5477849Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164131.xml
2022-09-27T16:41:38.4249135Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-09-27 16:41:38.424316]
2022-09-27T16:41:38.4250593Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:41:38.424433]
2022-09-27T16:41:40.4077754Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-09-27T16:41:40.4095922Z 
2022-09-27T16:41:40.4096196Z Running tests...
2022-09-27T16:41:40.4096616Z ----------------------------------------------------------------------
2022-09-27T16:41:42.1462170Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:42.1649039Z ok (1.755s)
2022-09-27T16:41:42.1674040Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-09-27T16:41:42.1740449Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s)
2022-09-27T16:41:42.1777994Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55136
2022-09-27T16:41:42.1784911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55137
2022-09-27T16:41:43.9404523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:43.9405059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:43.9406588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:43.9407081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:43.9415398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:43.9416140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:43.9419819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:43.9420594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:44.2066410Z dist init r=0, world=2
2022-09-27T16:41:44.2070744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:44.2268735Z dist init r=1, world=2
2022-09-27T16:41:44.2275251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:44.2276100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:44.2276809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:45.7753816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:45.7754387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:46.2717331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:46.2761300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:46.2762926Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:46.2763688Z   warnings.warn(
2022-09-27T16:41:46.2764814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:46.2765581Z   warnings.warn(
2022-09-27T16:41:46.3357482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:46.3358485Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:46.3360725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:46.3361357Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:46.3463480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:46.3463993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:46.9897131Z ok (4.815s)
2022-09-27T16:41:46.9919744Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55221
2022-09-27T16:41:46.9926068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55222
2022-09-27T16:41:48.6667258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:48.6667869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:48.6668760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:48.6669262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:48.7035569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:48.7036027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:48.7039853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:48.7040335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:48.9487342Z dist init r=0, world=2
2022-09-27T16:41:48.9491308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:48.9704180Z dist init r=1, world=2
2022-09-27T16:41:48.9710434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:48.9711317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:48.9797428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:50.5485343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:50.5485907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:51.0335153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.0335766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.0374326Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:51.0375229Z   warnings.warn(
2022-09-27T16:41:51.0376681Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:51.0377478Z   warnings.warn(
2022-09-27T16:41:51.0895791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:51.0896497Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:51.0898585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:51.0899209Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:51.0956017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.0956517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.8037974Z ok (4.814s)
2022-09-27T16:41:51.8062058Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55306
2022-09-27T16:41:51.8068562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55307
2022-09-27T16:41:53.5203489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:53.5203992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:53.5205808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:53.5206255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:53.5387633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:53.5388056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:53.5392343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:53.5392768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:53.8103345Z dist init r=1, world=2
2022-09-27T16:41:53.8109353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:53.8183388Z dist init r=0, world=2
2022-09-27T16:41:53.8189726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:53.8190619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:53.8211887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:55.3738967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:55.3739528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:55.8689250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:55.8690123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:55.8732641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:55.8733731Z   warnings.warn(
2022-09-27T16:41:55.8734948Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:41:55.8735814Z   warnings.warn(
2022-09-27T16:41:55.9290061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:55.9290760Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:55.9291946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:41:55.9292529Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:41:55.9391708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:55.9392280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:56.6179807Z ok (4.814s)
2022-09-27T16:41:56.6202500Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55391
2022-09-27T16:41:56.6208902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55392
2022-09-27T16:41:58.3283523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:58.3284096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:58.3285288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:58.3286063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:58.3555902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:58.3556388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:58.3560482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:58.3560973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:58.6075665Z dist init r=1, world=2
2022-09-27T16:41:58.6079961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:58.6251930Z dist init r=0, world=2
2022-09-27T16:41:58.6258477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:58.6259357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:58.6284035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:00.2083059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:00.2083561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:00.6815122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:00.6815674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:00.6853482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:00.6854330Z   warnings.warn(
2022-09-27T16:42:00.6855468Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:00.6856224Z   warnings.warn(
2022-09-27T16:42:00.7379625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:00.7380294Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:00.7383511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:00.7384162Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:00.7442773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:00.7443291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:01.4325048Z ok (4.814s)
2022-09-27T16:42:01.4348525Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55476
2022-09-27T16:42:01.4355207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55477
2022-09-27T16:42:03.1679809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:03.1680355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:03.1681834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:03.1682313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:03.1979694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:03.1980207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:03.1984582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:03.1985060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:03.4496886Z dist init r=1, world=2
2022-09-27T16:42:03.4502404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:03.4626929Z dist init r=0, world=2
2022-09-27T16:42:03.4633049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:03.4634299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:03.4706857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:05.0104126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:05.0104715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:05.5104621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:05.5151955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:05.5153467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:05.5154274Z   warnings.warn(
2022-09-27T16:42:05.5155422Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:05.5156173Z   warnings.warn(
2022-09-27T16:42:05.5264629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:05.5265329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:05.5309108Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5310452Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5312037Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5313255Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5314499Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5315743Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5316959Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.5318313Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:05.6076666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:05.6077401Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:05.6078884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:05.6079752Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:05.6179395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:05.6180180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:06.2480996Z ok (4.815s)
2022-09-27T16:42:06.2504176Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55561
2022-09-27T16:42:06.2510285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55562
2022-09-27T16:42:07.9841384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:07.9841897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:07.9843129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:07.9843625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:08.0181014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:08.0181511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:08.0185606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:08.0186101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:08.2708718Z dist init r=1, world=2
2022-09-27T16:42:08.2713273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:08.2860210Z dist init r=0, world=2
2022-09-27T16:42:08.2866037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:08.2867109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:08.2917634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:09.8565485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:09.8566053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:10.3341924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:10.3342494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:10.3381235Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:10.3382203Z   warnings.warn(
2022-09-27T16:42:10.3383335Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:10.3384083Z   warnings.warn(
2022-09-27T16:42:10.3490528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:10.3491090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:10.3532628Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3533923Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3535174Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3536402Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3537815Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3539046Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3540279Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.3541506Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:10.4365039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:10.4365714Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:10.4369124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:10.4369996Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:10.4428408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:10.4428897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:11.0620578Z ok (4.814s)
2022-09-27T16:42:11.0645933Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55646
2022-09-27T16:42:11.0653963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55647
2022-09-27T16:42:12.7453010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:12.7453574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:12.7455874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:12.7456338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:12.7789800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:12.7790277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:12.7794700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:12.7795168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:13.0208517Z dist init r=1, world=2
2022-09-27T16:42:13.0213666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:13.0448532Z dist init r=0, world=2
2022-09-27T16:42:13.0454739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:13.0455560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:13.0519178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:14.5368462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:14.5369084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:15.0006608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.0007143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.0053329Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:15.0054150Z   warnings.warn(
2022-09-27T16:42:15.0055480Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:15.0056262Z   warnings.warn(
2022-09-27T16:42:15.0162449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.0162969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.0204796Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0206022Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0207265Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0208514Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0209984Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0211250Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0212656Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0213895Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:15.0954989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:15.0955740Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:15.0956689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:15.0957331Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:15.1058337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.1058867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:15.7759338Z ok (4.714s)
2022-09-27T16:42:15.7784223Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55731
2022-09-27T16:42:15.7790396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55732
2022-09-27T16:42:17.5237051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:17.5237589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:17.5239145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:17.5239638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:17.5875649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:17.5888656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:17.5889360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:17.5890170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:17.7911422Z dist init r=1, world=2
2022-09-27T16:42:17.7915793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:17.8444254Z dist init r=0, world=2
2022-09-27T16:42:17.8450280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:17.8451439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:17.8525073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:19.3506367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:19.3506897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:19.8360775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:19.8361367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:19.8398842Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:19.8399667Z   warnings.warn(
2022-09-27T16:42:19.8400822Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:42:19.8401600Z   warnings.warn(
2022-09-27T16:42:19.8509690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:19.8510185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:19.8550699Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8552249Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8553534Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8554776Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8556019Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8557265Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8558505Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.8559710Z [W python_variable.cpp:326] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-09-27T16:42:19.9349763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:19.9350464Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:19.9354624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:19.9355315Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:19.9413725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:19.9414221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:20.5899629Z ok (4.814s)
2022-09-27T16:42:20.5899909Z 
2022-09-27T16:42:20.5900339Z ----------------------------------------------------------------------
2022-09-27T16:42:20.5900625Z Ran 11 tests in 40.180s
2022-09-27T16:42:20.5900800Z 
2022-09-27T16:42:20.5900899Z OK
2022-09-27T16:42:20.5901015Z 
2022-09-27T16:42:20.5901168Z Generating XML reports...
2022-09-27T16:42:20.5964276Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220927164140.xml
2022-09-27T16:42:20.5977492Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220927164140.xml
2022-09-27T16:42:21.0412985Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-09-27 16:42:21.040743]
2022-09-27T16:42:21.0413816Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:42:21.040841]
2022-09-27T16:42:22.9430632Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2022-09-27T16:42:22.9446483Z 
2022-09-27T16:42:22.9446628Z Running tests...
2022-09-27T16:42:22.9447283Z ----------------------------------------------------------------------
2022-09-27T16:42:22.9454681Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:42:24.5000264Z This unit test verifies the ``allreduce`` hook registered case gives same result ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:24.5806301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55851
2022-09-27T16:42:24.5811027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55852
2022-09-27T16:42:26.2301949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:26.2302523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:26.2303688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:26.2304195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:26.3044445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:26.3044942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:26.3048059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:26.3048580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:26.4720798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:26.5405556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:27.8754087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rwp3d9h
2022-09-27T16:42:27.8755065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rwp3d9h/_remote_module_non_scriptable.py
2022-09-27T16:42:27.9582500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7die50wb
2022-09-27T16:42:27.9584497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7die50wb/_remote_module_non_scriptable.py
2022-09-27T16:42:29.1564485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:29.1565221Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:29.1566203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:29.1566868Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:29.6924969Z ok (6.747s)
2022-09-27T16:42:29.6930954Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:42:29.6946532Z This unit test verifies the ``fp16 compress`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55937
2022-09-27T16:42:29.6953033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55938
2022-09-27T16:42:31.3408611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:31.3409208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:31.3410163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:31.3410879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:31.3776559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:31.3777015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:31.3780279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:31.3780755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:31.5949280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:31.6183489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:33.0182489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrjh68d0
2022-09-27T16:42:33.0183108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrjh68d0/_remote_module_non_scriptable.py
2022-09-27T16:42:33.0901423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvkl52to
2022-09-27T16:42:33.0902699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvkl52to/_remote_module_non_scriptable.py
2022-09-27T16:42:34.1702408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:34.1703238Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:34.1704165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:34.1705220Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:34.7064312Z ok (5.014s)
2022-09-27T16:42:34.7070432Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:42:34.7086041Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56023
2022-09-27T16:42:34.7093045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56024
2022-09-27T16:42:36.3672416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:36.3672949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:36.3674316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:36.3674844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:36.3818429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:36.3818916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:36.3822237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:36.3822735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:36.6229669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:36.6296791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:38.0604502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppw6ghpvw
2022-09-27T16:42:38.0605594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppw6ghpvw/_remote_module_non_scriptable.py
2022-09-27T16:42:38.0948088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4f9j9cj
2022-09-27T16:42:38.0948958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4f9j9cj/_remote_module_non_scriptable.py
2022-09-27T16:42:39.2321782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:39.2322557Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:39.2323533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:39.2324217Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:39.8224953Z ok (5.116s)
2022-09-27T16:42:39.8229910Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:42:39.8245816Z This unit test verifies the ``quantize per channel`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56109
2022-09-27T16:42:39.8252483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56110
2022-09-27T16:42:41.4579389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:41.4579945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:41.4580917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:41.4581780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:41.4912959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:41.4913477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:41.4916455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:41.4916943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:41.7069408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:41.7312075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:43.1425066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpite28zfk
2022-09-27T16:42:43.1426276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpite28zfk/_remote_module_non_scriptable.py
2022-09-27T16:42:43.1950504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrjo2_8_
2022-09-27T16:42:43.1953004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrjo2_8_/_remote_module_non_scriptable.py
2022-09-27T16:42:44.3136161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:44.3136953Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:44.3138251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:44.3138965Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:44.9366973Z ok (5.114s)
2022-09-27T16:42:44.9372527Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2022-09-27T16:42:44.9389549Z This unit test verifies the ``quantize per tensor`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56195
2022-09-27T16:42:44.9396637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56196
2022-09-27T16:42:46.6325648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:46.6326183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:46.6327021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:46.6327511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:46.6500523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:46.6500996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:46.6504220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:46.6504689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:46.8913197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:46.8997938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:48.3400504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeeph1i_6
2022-09-27T16:42:48.3401138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeph1i_6/_remote_module_non_scriptable.py
2022-09-27T16:42:48.3585035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtlt31yh
2022-09-27T16:42:48.3588612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtlt31yh/_remote_module_non_scriptable.py
2022-09-27T16:42:49.5485897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:49.5486658Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:49.5497146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-09-27T16:42:49.5497798Z   warnings.warn(msg, FutureWarning)
2022-09-27T16:42:50.1506914Z ok (5.214s)
2022-09-27T16:42:50.1534213Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56281
2022-09-27T16:42:50.1541339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56282
2022-09-27T16:42:51.8104371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:51.8104928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:51.8105794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:51.8106298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:51.8311917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:51.8312420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:51.8315456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:51.8315932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:52.0598428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:52.0725554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:55.1528819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperg5jxzm
2022-09-27T16:42:55.1529472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperg5jxzm/_remote_module_non_scriptable.py
2022-09-27T16:42:55.1869820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxml1mvn
2022-09-27T16:42:55.1870891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxml1mvn/_remote_module_non_scriptable.py
2022-09-27T16:42:57.6701144Z ok (7.519s)
2022-09-27T16:42:57.6701377Z 
2022-09-27T16:42:57.6701794Z ----------------------------------------------------------------------
2022-09-27T16:42:57.6702149Z Ran 6 tests in 34.725s
2022-09-27T16:42:57.6702321Z 
2022-09-27T16:42:57.6702424Z OK
2022-09-27T16:42:57.6702565Z 
2022-09-27T16:42:57.6702708Z Generating XML reports...
2022-09-27T16:42:57.6748224Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220927164222.xml
2022-09-27T16:42:58.0965371Z Running distributed/test_c10d_common ... [2022-09-27 16:42:58.095954]
2022-09-27T16:42:58.0966181Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:42:58.096058]
2022-09-27T16:43:00.0469358Z test_debug_level (__main__.CommTest)
2022-09-27T16:43:00.0470076Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-09-27T16:43:00.0470809Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-09-27T16:43:00.0471226Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-09-27T16:43:00.0471680Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-09-27T16:43:00.0472424Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-09-27T16:43:00.0473235Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-09-27T16:43:00.0474174Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-09-27T16:43:00.0474764Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-09-27T16:43:01.6668999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:01.6669547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:01.6671320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:01.6671898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:01.9370615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:01.9384275Z 
2022-09-27T16:43:01.9384472Z Running tests...
2022-09-27T16:43:01.9385101Z ----------------------------------------------------------------------
2022-09-27T16:43:03.5555126Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:03.6412684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56435
2022-09-27T16:43:03.6417811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56436
2022-09-27T16:43:05.3308042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:05.3309092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:05.3310250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:05.3311258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:05.3688388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:05.3689292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:05.3692906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:05.3693858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:05.6067425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:05.6308135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:06.0479940Z ok (4.109s)
2022-09-27T16:43:06.0480206Z 
2022-09-27T16:43:06.0480639Z ----------------------------------------------------------------------
2022-09-27T16:43:06.0480946Z Ran 1 test in 4.109s
2022-09-27T16:43:06.0481132Z 
2022-09-27T16:43:06.0481244Z OK
2022-09-27T16:43:06.0481391Z 
2022-09-27T16:43:06.0481534Z Generating XML reports...
2022-09-27T16:43:06.0520881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220927164301.xml
2022-09-27T16:43:08.1378986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:08.1379459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:08.1380921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:08.1381472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:08.3942501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:08.3955607Z 
2022-09-27T16:43:08.3955968Z Running tests...
2022-09-27T16:43:08.3956438Z ----------------------------------------------------------------------
2022-09-27T16:43:09.9936091Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:10.0779096Z ok (1.682s)
2022-09-27T16:43:10.0779361Z 
2022-09-27T16:43:10.0779855Z ----------------------------------------------------------------------
2022-09-27T16:43:10.0780212Z Ran 1 test in 1.682s
2022-09-27T16:43:10.0780366Z 
2022-09-27T16:43:10.0780441Z OK
2022-09-27T16:43:10.0780582Z 
2022-09-27T16:43:10.0780696Z Generating XML reports...
2022-09-27T16:43:10.0814513Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164308.xml
2022-09-27T16:43:12.1464259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:12.1465300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:12.1466516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:12.1467448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:12.4141429Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:12.4156307Z 
2022-09-27T16:43:12.4156801Z Running tests...
2022-09-27T16:43:12.4157345Z ----------------------------------------------------------------------
2022-09-27T16:43:14.0766596Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:14.1602409Z ok (1.744s)
2022-09-27T16:43:14.1602661Z 
2022-09-27T16:43:14.1603418Z ----------------------------------------------------------------------
2022-09-27T16:43:14.1603743Z Ran 1 test in 1.744s
2022-09-27T16:43:14.1603918Z 
2022-09-27T16:43:14.1604038Z OK
2022-09-27T16:43:14.1604182Z 
2022-09-27T16:43:14.1604322Z Generating XML reports...
2022-09-27T16:43:14.1638736Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164312.xml
2022-09-27T16:43:16.2334048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:16.2335095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:16.2337206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:16.2338144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:16.4915044Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:16.4928343Z 
2022-09-27T16:43:16.4928600Z Running tests...
2022-09-27T16:43:16.4929055Z ----------------------------------------------------------------------
2022-09-27T16:43:18.0985176Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:18.1836685Z ok (1.690s)
2022-09-27T16:43:18.1836986Z 
2022-09-27T16:43:18.1837454Z ----------------------------------------------------------------------
2022-09-27T16:43:18.1837812Z Ran 1 test in 1.691s
2022-09-27T16:43:18.1837953Z 
2022-09-27T16:43:18.1838043Z OK
2022-09-27T16:43:18.1838173Z 
2022-09-27T16:43:18.1838267Z Generating XML reports...
2022-09-27T16:43:18.1875463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164316.xml
2022-09-27T16:43:20.3044344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:20.3044878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:20.3047019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:20.3047788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:20.5751669Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:20.5765562Z 
2022-09-27T16:43:20.5765943Z Running tests...
2022-09-27T16:43:20.5766404Z ----------------------------------------------------------------------
2022-09-27T16:43:22.1938048Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:22.2801294Z ok (1.703s)
2022-09-27T16:43:22.2801476Z 
2022-09-27T16:43:22.2801934Z ----------------------------------------------------------------------
2022-09-27T16:43:22.2802291Z Ran 1 test in 1.703s
2022-09-27T16:43:22.2802464Z 
2022-09-27T16:43:22.2802576Z OK
2022-09-27T16:43:22.2802716Z 
2022-09-27T16:43:22.2802831Z Generating XML reports...
2022-09-27T16:43:22.2836775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164320.xml
2022-09-27T16:43:24.4031555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:24.4032050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:24.4033776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:24.4034265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:24.6722443Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:24.6737246Z 
2022-09-27T16:43:24.6737718Z Running tests...
2022-09-27T16:43:24.6738684Z ----------------------------------------------------------------------
2022-09-27T16:43:26.3114055Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:26.4029197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56674
2022-09-27T16:43:26.4033810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56675
2022-09-27T16:43:26.4040832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56676
2022-09-27T16:43:26.4047832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56677
2022-09-27T16:43:28.0945679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.0946258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.0947765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.0948247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.1086563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.1087035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.1090911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.1091372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.1467229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.1467701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.1471807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.1472256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.1646527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.1647292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.1651139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.1651569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.3952314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:28.3978709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:43:28.4209044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:28.4383448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:43:29.0126274Z ok (4.338s)
2022-09-27T16:43:29.0126693Z 
2022-09-27T16:43:29.0127361Z ----------------------------------------------------------------------
2022-09-27T16:43:29.0127976Z Ran 1 test in 4.339s
2022-09-27T16:43:29.0128261Z 
2022-09-27T16:43:29.0128384Z OK
2022-09-27T16:43:29.0128623Z 
2022-09-27T16:43:29.0128859Z Generating XML reports...
2022-09-27T16:43:29.0170363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164324.xml
2022-09-27T16:43:31.1500619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:31.1501173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:31.1502279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:31.1502763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:31.4147620Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:31.4160838Z 
2022-09-27T16:43:31.4161191Z Running tests...
2022-09-27T16:43:31.4161677Z ----------------------------------------------------------------------
2022-09-27T16:43:33.0561222Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:33.1476568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56845
2022-09-27T16:43:33.1481603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56846
2022-09-27T16:43:33.1488125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56847
2022-09-27T16:43:33.1495176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56848
2022-09-27T16:43:34.8295532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:34.8296581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:34.8297751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:34.8298732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:34.8404986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:34.8405866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:34.8411098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:34.8412088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:34.8539809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:34.8540766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:34.8544387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:34.8545796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:34.8853951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:34.8854937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:34.8857451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:34.8858458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:35.1268934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:43:35.1275594Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-09-27T16:43:35.1285999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:35.1294773Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-09-27T16:43:35.1305000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:43:35.1312055Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-09-27T16:43:35.1550977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:36.1288817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:43:36.1305771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:36.1322545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:43:36.1360127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:36.1361526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:36.1392861Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:36.1409176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:36.1426396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:38.6633318Z ok (7.247s)
2022-09-27T16:43:38.6633580Z 
2022-09-27T16:43:38.6634024Z ----------------------------------------------------------------------
2022-09-27T16:43:38.6634379Z Ran 1 test in 7.247s
2022-09-27T16:43:38.6634558Z 
2022-09-27T16:43:38.6634664Z OK
2022-09-27T16:43:38.6634806Z 
2022-09-27T16:43:38.6634957Z Generating XML reports...
2022-09-27T16:43:38.6674762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164331.xml
2022-09-27T16:43:40.7859531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:40.7860061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:40.7861155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:40.7861628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:41.0464310Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:41.0477849Z 
2022-09-27T16:43:41.0478011Z Running tests...
2022-09-27T16:43:41.0478494Z ----------------------------------------------------------------------
2022-09-27T16:43:42.6427841Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:42.7382013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57025
2022-09-27T16:43:42.7387780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57026
2022-09-27T16:43:42.7394877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57027
2022-09-27T16:43:42.7401976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57028
2022-09-27T16:43:44.4025720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:44.4026695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:44.4027910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:44.4028938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:44.4219772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:44.4220643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:44.4226022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:44.4226996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:44.4743352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:44.4744368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:44.4746104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:44.4747494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:44.4750623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:44.4751543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:44.4755317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:44.4756299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:44.6897778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:43:44.6920773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:43:44.7534450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:44.7552019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:45.3478892Z ok (4.300s)
2022-09-27T16:43:45.3479141Z 
2022-09-27T16:43:45.3479554Z ----------------------------------------------------------------------
2022-09-27T16:43:45.3479878Z Ran 1 test in 4.300s
2022-09-27T16:43:45.3480054Z 
2022-09-27T16:43:45.3480159Z OK
2022-09-27T16:43:45.3480302Z 
2022-09-27T16:43:45.3480447Z Generating XML reports...
2022-09-27T16:43:45.3520508Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164341.xml
2022-09-27T16:43:47.4286040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:47.4286579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:47.4287777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:47.4288234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:47.6870247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-09-27T16:43:47.6883827Z 
2022-09-27T16:43:47.6884129Z Running tests...
2022-09-27T16:43:47.6884833Z ----------------------------------------------------------------------
2022-09-27T16:43:49.2887980Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:49.3736277Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57196
2022-09-27T16:43:49.3741881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57197
2022-09-27T16:43:49.3748643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57198
2022-09-27T16:43:49.3754614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57199
2022-09-27T16:43:51.0921838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:51.0922413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:51.0923848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:51.0924336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:51.1019259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:51.1019743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:51.1023791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:51.1024269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:51.1072577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:51.1073384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:51.1077342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:51.1077831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:51.1410929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:51.1411406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:51.1415222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:51.1415705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:51.3912130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:51.4056737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:51.4067114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:43:51.4068005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:51.4078267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-09-27T16:43:51.4146791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:43:51.4157706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-09-27T16:43:51.4231560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:51.4232635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:51.4261073Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:51.4273101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:51.4283310Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-09-27T16:43:53.9881046Z ok (6.299s)
2022-09-27T16:43:53.9881368Z 
2022-09-27T16:43:53.9881820Z ----------------------------------------------------------------------
2022-09-27T16:43:53.9882159Z Ran 1 test in 6.300s
2022-09-27T16:43:53.9882284Z 
2022-09-27T16:43:53.9882389Z OK
2022-09-27T16:43:53.9882532Z 
2022-09-27T16:43:53.9882676Z Generating XML reports...
2022-09-27T16:43:53.9921522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164347.xml
2022-09-27T16:43:54.7422557Z Running distributed/elastic/multiprocessing/api_test ... [2022-09-27 16:43:54.741716]
2022-09-27T16:43:54.7423416Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:43:54.741827]
2022-09-27T16:43:56.8099476Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-09-27T16:43:56.8120430Z 
2022-09-27T16:43:56.8120797Z Running tests...
2022-09-27T16:43:56.8121317Z ----------------------------------------------------------------------
2022-09-27T16:43:58.5504399Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.738s)
2022-09-27T16:43:58.5514241Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-09-27T16:43:58.5532597Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-09-27T16:43:58.5874898Z   test_binary (__main__.StartProcessesListTest) ... hello stdout from 0
2022-09-27T16:43:58.5875269Z hello stderr from 0
2022-09-27T16:43:58.5880871Z hello stdout from 1
2022-09-27T16:43:58.5881156Z hello stderr from 1
2022-09-27T16:43:58.6579267Z ok (0.104s)
2022-09-27T16:43:58.6916002Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1
2022-09-27T16:43:58.6916333Z bar stderr from 1
2022-09-27T16:43:58.7622049Z failed (exitcode: 138) local_rank: 0 (pid: 57377) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-09-27T16:43:58.7632599Z ok (0.105s)
2022-09-27T16:43:58.7662105Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.003s)
2022-09-27T16:43:58.7995255Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-09-27T16:43:58.7995792Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-09-27T16:43:58.7996233Z     raise RuntimeError(f"raised from {rank}")
2022-09-27T16:43:58.7996528Z RuntimeError: raised from 0
2022-09-27T16:43:58.7996806Z bar from 1
2022-09-27T16:43:58.8694982Z failed (exitcode: 1) local_rank: 0 (pid: 57380) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-09-27T16:43:58.8701378Z ok (0.104s)
2022-09-27T16:43:58.9036498Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-09-27T16:43:58.9743031Z [trainer1]:world stderr from 1
2022-09-27T16:43:58.9743280Z [trainer0]:hello stdout from 0
2022-09-27T16:43:59.9772748Z ok (1.107s)
2022-09-27T16:44:01.8079386Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:01.8079967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:01.8089835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:01.8090546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:01.8310554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:01.8310971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:01.8326327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:01.8326780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:02.0967021Z hello stdout from 0
2022-09-27T16:44:02.0967341Z hello stderr from 0
2022-09-27T16:44:02.1023581Z hello stdout from 1
2022-09-27T16:44:02.1023841Z hello stderr from 1
2022-09-27T16:44:02.5876937Z Closing process 57388 via signal SIGTERM
2022-09-27T16:44:02.5991394Z ok (2.622s)
2022-09-27T16:44:04.4650306Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.4651466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.4659953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.4660849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.4661905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.4662725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:04.4673492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.4674419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.4675613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.4676464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:04.4687929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.4688863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:04.4825618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.4827504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.4840225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.4841191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:05.5926304Z Closing process 57453 via signal SIGTERM
2022-09-27T16:44:05.5931879Z Closing process 57454 via signal SIGTERM
2022-09-27T16:44:05.6311735Z ok (3.032s)
2022-09-27T16:44:05.6325518Z   test_function_raise (__main__.StartProcessesListTest)
2022-09-27T16:44:07.4685203Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:07.4685857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:07.4690801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:07.4691364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:07.4698580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:07.4699110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:07.4704986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:07.4705556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:08.0533058Z failed (exitcode: 1) local_rank: 0 (pid: 57585) of fn: echo2 (start_method: spawn)
2022-09-27T16:44:08.0533487Z Traceback (most recent call last):
2022-09-27T16:44:08.0534215Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-09-27T16:44:08.0535022Z     self._pc.join(-1)
2022-09-27T16:44:08.0535565Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-09-27T16:44:08.0536047Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-09-27T16:44:08.0536623Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-09-27T16:44:08.0536912Z 
2022-09-27T16:44:08.0537154Z -- Process 0 terminated with the following error:
2022-09-27T16:44:08.0537471Z Traceback (most recent call last):
2022-09-27T16:44:08.0538016Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-09-27T16:44:08.0538410Z     fn(i, *args)
2022-09-27T16:44:08.0538946Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-09-27T16:44:08.0539382Z     ret = record(fn)(*args_)
2022-09-27T16:44:08.0539994Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-09-27T16:44:08.0540446Z     return f(*args, **kwargs)
2022-09-27T16:44:08.0540854Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-09-27T16:44:08.0541277Z     raise RuntimeError(msg)
2022-09-27T16:44:08.0541574Z RuntimeError: hello
2022-09-27T16:44:08.0541727Z 
2022-09-27T16:44:08.0550896Z ok (2.424s)
2022-09-27T16:44:08.0577239Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-09-27T16:44:08.0592018Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2022-09-27T16:44:08.0621291Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 57651 via signal SIGTERM
2022-09-27T16:44:08.0628052Z ok (0.004s)
2022-09-27T16:44:08.0665832Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-09-27T16:44:08.0666268Z Traceback (most recent call last):
2022-09-27T16:44:08.0666841Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-09-27T16:44:08.0667287Z     self._pc.join(-1)
2022-09-27T16:44:08.0667634Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-09-27T16:44:08.0667975Z     return self._mock_call(*args, **kwargs)
2022-09-27T16:44:08.0668348Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-09-27T16:44:08.0668952Z     return self._execute_mock_call(*args, **kwargs)
2022-09-27T16:44:08.0669327Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-09-27T16:44:08.0669680Z     raise effect
2022-09-27T16:44:08.0670056Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-09-27T16:44:08.0673075Z ok (0.004s)
2022-09-27T16:44:09.8598717Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:09.8599334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:09.8609485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:09.8610330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:11.4626394Z ok (3.395s)
2022-09-27T16:44:11.4654702Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 57685 closing signal SIGTERM
2022-09-27T16:44:11.4670092Z ok (0.004s)
2022-09-27T16:44:11.4687095Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-09-27T16:44:11.4696842Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-09-27T16:44:13.2860283Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:13.2861218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:13.2872994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:13.2873448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:13.3149333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:13.3149820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:13.3165244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:13.3165972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:13.5410198Z world
2022-09-27T16:44:13.5582907Z hello
2022-09-27T16:44:13.9651608Z Closing process 57686 via signal SIGTERM
2022-09-27T16:44:13.9765747Z ok (2.507s)
2022-09-27T16:44:13.9787842Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-09-27T16:44:14.0136356Z   test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1
2022-09-27T16:44:14.0136692Z bar stderr from 1
2022-09-27T16:44:14.0826974Z failed (exitcode: 138) local_rank: 0 (pid: 57752) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-09-27T16:44:14.0835849Z ok (0.105s)
2022-09-27T16:44:14.0860700Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2022-09-27T16:44:14.1185108Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-09-27T16:44:14.1185870Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-09-27T16:44:14.1186284Z     raise RuntimeError(f"raised from {rank}")
2022-09-27T16:44:14.1186587Z RuntimeError: raised from 0
2022-09-27T16:44:14.1197001Z bar from 1
2022-09-27T16:44:14.1891807Z failed (exitcode: 1) local_rank: 0 (pid: 57755) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-09-27T16:44:14.1898095Z ok (0.104s)
2022-09-27T16:44:16.0572495Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:16.0573094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:16.0586600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:16.0587134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:16.0886967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:16.0887468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:16.0901893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:16.0902343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:16.1224057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:16.1224540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:16.1239387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:16.1239856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:16.1672987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:16.1673496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:16.1690009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:16.1690688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:17.3001135Z Closing process 57757 via signal SIGTERM
2022-09-27T16:44:17.3009540Z Closing process 57758 via signal SIGTERM
2022-09-27T16:44:17.3385078Z ok (3.148s)
2022-09-27T16:44:17.3398595Z   test_function_raise (__main__.StartProcessesTest)
2022-09-27T16:44:19.1076263Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:19.1077465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:19.1091987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:19.1093002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:19.1378931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:19.1379959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:19.1398291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:19.1399292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:19.7609528Z failed (exitcode: 1) local_rank: 0 (pid: 57889) of fn: echo2 (start_method: spawn)
2022-09-27T16:44:19.7610511Z Traceback (most recent call last):
2022-09-27T16:44:19.7611576Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-09-27T16:44:19.7612055Z     self._pc.join(-1)
2022-09-27T16:44:19.7612542Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-09-27T16:44:19.7613030Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-09-27T16:44:19.7613472Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-09-27T16:44:19.7613746Z 
2022-09-27T16:44:19.7613985Z -- Process 0 terminated with the following error:
2022-09-27T16:44:19.7614322Z Traceback (most recent call last):
2022-09-27T16:44:19.7614810Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-09-27T16:44:19.7615176Z     fn(i, *args)
2022-09-27T16:44:19.7615692Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-09-27T16:44:19.7616093Z     ret = record(fn)(*args_)
2022-09-27T16:44:19.7616630Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-09-27T16:44:19.7617049Z     return f(*args, **kwargs)
2022-09-27T16:44:19.7617456Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-09-27T16:44:19.7617823Z     raise RuntimeError(msg)
2022-09-27T16:44:19.7618103Z RuntimeError: hello
2022-09-27T16:44:19.7618274Z 
2022-09-27T16:44:19.7627889Z ok (2.424s)
2022-09-27T16:44:19.7654316Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-09-27T16:44:19.7668764Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2022-09-27T16:44:19.7698511Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 57955 via signal SIGTERM
2022-09-27T16:44:19.7705985Z ok (0.004s)
2022-09-27T16:44:19.7738145Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-09-27T16:44:19.7738632Z Traceback (most recent call last):
2022-09-27T16:44:19.7739187Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-09-27T16:44:19.7739836Z     self._pc.join(-1)
2022-09-27T16:44:19.7740159Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-09-27T16:44:19.7740519Z     return self._mock_call(*args, **kwargs)
2022-09-27T16:44:19.7740893Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-09-27T16:44:19.7741272Z     return self._execute_mock_call(*args, **kwargs)
2022-09-27T16:44:19.7741650Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-09-27T16:44:19.7742004Z     raise effect
2022-09-27T16:44:19.7742389Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-09-27T16:44:19.7744164Z ok (0.004s)
2022-09-27T16:44:21.5495409Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:21.5507555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:21.5508841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:21.5509788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:23.1751683Z ok (3.400s)
2022-09-27T16:44:23.1775244Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 57989 closing signal SIGTERM
2022-09-27T16:44:23.1789818Z ok (0.004s)
2022-09-27T16:44:23.1805268Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2022-09-27T16:44:23.1814597Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-09-27T16:44:25.0290895Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:25.0291535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:25.0302826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:25.0303336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:25.0480157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:25.0480642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:25.0497204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:25.0497699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:25.2970897Z hello
2022-09-27T16:44:25.3044788Z world
2022-09-27T16:44:25.7910228Z Closing process 57991 via signal SIGTERM
2022-09-27T16:44:25.8020318Z ok (2.620s)
2022-09-27T16:44:25.8036969Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-09-27T16:44:25.8048318Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-09-27T16:44:25.8059001Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-09-27T16:44:25.8059441Z 
2022-09-27T16:44:25.8060014Z ----------------------------------------------------------------------
2022-09-27T16:44:25.8060381Z Ran 38 tests in 28.994s
2022-09-27T16:44:25.8060559Z 
2022-09-27T16:44:25.8060668Z OK
2022-09-27T16:44:25.8060813Z 
2022-09-27T16:44:25.8060924Z Generating XML reports...
2022-09-27T16:44:25.8100884Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220927164356.xml
2022-09-27T16:44:25.8124251Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220927164356.xml
2022-09-27T16:44:25.8145033Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220927164356.xml
2022-09-27T16:44:25.8149834Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220927164356.xml
2022-09-27T16:44:26.2696212Z Running distributed/_shard/checkpoint/test_checkpoint ... [2022-09-27 16:44:26.269020]
2022-09-27T16:44:26.2697661Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:44:26.269122]
2022-09-27T16:44:28.2691753Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint
2022-09-27T16:44:28.2711494Z 
2022-09-27T16:44:28.2711787Z Running tests...
2022-09-27T16:44:28.2712236Z ----------------------------------------------------------------------
2022-09-27T16:44:29.9976057Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:30.0170340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58091
2022-09-27T16:44:30.0176754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58092
2022-09-27T16:44:31.7620094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:31.7620675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:31.7621335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:31.7621794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:31.7765622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:31.7766131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:31.7769968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:31.7770551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:32.0295406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:32.0299176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:32.0516122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:32.0517954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:32.0518890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:32.0618606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:34.2279725Z ok (5.956s)
2022-09-27T16:44:34.2304217Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58172
2022-09-27T16:44:34.2310454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58173
2022-09-27T16:44:35.9658899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:35.9659454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:35.9660058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:35.9660544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:36.0037335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:36.0037863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:36.0041339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:36.0042121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:36.2382436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:36.2550261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:36.2595092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:36.2766870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:36.2767723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:36.2799108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:38.4407576Z ok (4.213s)
2022-09-27T16:44:38.4430868Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58253
2022-09-27T16:44:38.4437267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58254
2022-09-27T16:44:38.4445615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58255
2022-09-27T16:44:38.4452510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58256
2022-09-27T16:44:40.2114214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:40.2114769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:40.2115392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:40.2116118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:40.2252697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:40.2253210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:40.2256437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:40.2256918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:40.2432981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:40.2433455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:40.2436250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:40.2436728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:40.2742466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:40.2743023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:40.2743597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:40.2744088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:40.4944917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:40.4959901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:40.5047001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:40.5192798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:41.0524908Z skip: Need at least 4 CUDA devices (2.612s)
2022-09-27T16:44:41.0548396Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58389
2022-09-27T16:44:41.0554966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58390
2022-09-27T16:44:41.0561684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58391
2022-09-27T16:44:41.0568857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58392
2022-09-27T16:44:42.7847790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.7848327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.7849763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.7850302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:42.7888188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.7888930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.7891821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.7892540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:42.8119129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.8119878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.8122766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.8123494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:42.8279422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.8279947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.8282875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.8283680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:43.0610921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:43.0721112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:43.0764222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:43.0826757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:43.6642374Z skip: Need at least 4 CUDA devices (2.612s)
2022-09-27T16:44:43.6669496Z   test_load_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58525
2022-09-27T16:44:43.6675859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58526
2022-09-27T16:44:43.6683091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58527
2022-09-27T16:44:43.6689599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58528
2022-09-27T16:44:45.4126913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:45.4127638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:45.4128324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:45.4129015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:45.4260686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:45.4261194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:45.4263996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:45.4264732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:45.4618580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:45.4619272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:45.4622559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:45.4623333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:45.4683145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:45.4683810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:45.4686967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:45.4687745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:45.7038897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:45.7071437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:45.7109767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:45.7220100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:46.2762243Z skip: Need at least 4 CUDA devices (2.612s)
2022-09-27T16:44:46.2786093Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58661
2022-09-27T16:44:46.2792427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58662
2022-09-27T16:44:46.2799201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58663
2022-09-27T16:44:46.2806258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58664
2022-09-27T16:44:48.0244583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.0245158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.0245743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.0246212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.0331338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.0331802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.0335286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.0335776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.0373056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.0373529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.0377086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.0377547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.0569891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.0570677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.0573881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.0574614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.3033548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:48.3034167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:48.3096997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:48.3224098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:48.8879825Z ok (2.612s)
2022-09-27T16:44:48.8906783Z   test_save_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58797
2022-09-27T16:44:48.8914008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58798
2022-09-27T16:44:48.8920422Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58799
2022-09-27T16:44:48.8927658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58800
2022-09-27T16:44:50.6261310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:50.6261861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:50.6262745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:50.6263253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:50.6304474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:50.6305276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:50.6308265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:50.6308781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:50.6448621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:50.6449092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:50.6452201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:50.6452693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:50.6604835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:50.6605306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:50.6608313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:50.6608824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:50.9060724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:50.9146825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:50.9150747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:50.9273006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:51.5010845Z skip: Need at least 4 CUDA devices (2.613s)
2022-09-27T16:44:51.5033765Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58933
2022-09-27T16:44:51.5040166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58934
2022-09-27T16:44:51.5047237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58935
2022-09-27T16:44:51.5054245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58936
2022-09-27T16:44:53.2337369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:53.2337911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:53.2338529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:53.2339008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:53.2339594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:53.2340990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:53.2343051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:53.2343548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:53.2716591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:53.2717072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:53.2719444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:53.2719903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:53.2969102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:53.2969991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:53.2970941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:53.2971430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:53.5088214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:44:53.5220523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:44:53.5321937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:53.5475126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:54.1136174Z ok (2.612s)
2022-09-27T16:44:54.1136434Z 
2022-09-27T16:44:54.1136870Z ----------------------------------------------------------------------
2022-09-27T16:44:54.1137184Z Ran 8 tests in 25.842s
2022-09-27T16:44:54.1137377Z 
2022-09-27T16:44:54.1137487Z OK (skipped=4)
2022-09-27T16:44:54.1137700Z 
2022-09-27T16:44:54.1137829Z Generating XML reports...
2022-09-27T16:44:54.1179129Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220927164428.xml
2022-09-27T16:44:54.1188536Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220927164428.xml
2022-09-27T16:44:54.5760854Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-09-27 16:44:54.575561]
2022-09-27T16:44:54.5761634Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:44:54.575667]
2022-09-27T16:44:56.4951093Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap
2022-09-27T16:44:56.4966528Z 
2022-09-27T16:44:56.4966768Z Running tests...
2022-09-27T16:44:56.4967284Z ----------------------------------------------------------------------
2022-09-27T16:44:56.4978420Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-09-27T16:44:58.1764690Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:58.1955903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59104
2022-09-27T16:44:58.1963015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59105
2022-09-27T16:44:59.8909100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:59.8909526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:59.8910138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:59.8910678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:59.9195576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:59.9196131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:59.9198691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:59.9199226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:00.1861438Z dist init r=0, world=2
2022-09-27T16:45:00.1866950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:00.1873344Z dist init r=1, world=2
2022-09-27T16:45:00.1879607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:00.1880476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:00.1970691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:01.7891411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:01.7892091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:01.8139371Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:01.8140363Z   warnings.warn(
2022-09-27T16:45:01.8183491Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:01.8184288Z   warnings.warn(
2022-09-27T16:45:02.9076336Z ok (6.410s)
2022-09-27T16:45:02.9085716Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-09-27T16:45:02.9101860Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59189
2022-09-27T16:45:02.9109014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59190
2022-09-27T16:45:04.6178704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:04.6179682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:04.6180385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:04.6181130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:04.6387002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:04.6387486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:04.6391235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:04.6391724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:04.9003595Z dist init r=1, world=2
2022-09-27T16:45:04.9008153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:04.9123714Z dist init r=0, world=2
2022-09-27T16:45:04.9130751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:04.9131527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:04.9213315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:06.4592068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:06.4592672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:06.4812435Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:06.4813279Z   warnings.warn(
2022-09-27T16:45:06.4814410Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:06.4815186Z   warnings.warn(
2022-09-27T16:45:07.5218908Z ok (4.614s)
2022-09-27T16:45:07.5228388Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-09-27T16:45:07.5245473Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59274
2022-09-27T16:45:07.5252661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59275
2022-09-27T16:45:09.2258754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:09.2259543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:09.2260177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:09.2260661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:09.2386282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:09.2386824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:09.2390964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:09.2391509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:09.5134504Z dist init r=1, world=2
2022-09-27T16:45:09.5140030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:09.5146705Z dist init r=0, world=2
2022-09-27T16:45:09.5153568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:09.5154369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:09.5243199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:11.0851072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:11.0851597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:11.1343857Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:11.1344806Z   warnings.warn(
2022-09-27T16:45:11.1345919Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:11.1346674Z   warnings.warn(
2022-09-27T16:45:11.7353277Z ok (4.213s)
2022-09-27T16:45:11.7362231Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-09-27T16:45:11.7379860Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59355
2022-09-27T16:45:11.7386481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59356
2022-09-27T16:45:13.4152366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:13.4152943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:13.4153873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:13.4154350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:13.4446842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:13.4447391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:13.4450822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:13.4451372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:13.6992745Z dist init r=1, world=2
2022-09-27T16:45:13.6998219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:13.7152280Z dist init r=0, world=2
2022-09-27T16:45:13.7158958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:13.7159772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:13.7202633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:15.2685656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:15.2686201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:15.3065077Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:15.3065871Z   warnings.warn(
2022-09-27T16:45:15.3066966Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:45:15.3067712Z   warnings.warn(
2022-09-27T16:45:15.8480536Z ok (4.113s)
2022-09-27T16:45:15.8480922Z 
2022-09-27T16:45:15.8481387Z ----------------------------------------------------------------------
2022-09-27T16:45:15.8481727Z Ran 4 tests in 19.351s
2022-09-27T16:45:15.8481901Z 
2022-09-27T16:45:15.8482001Z OK
2022-09-27T16:45:15.8482137Z 
2022-09-27T16:45:15.8482278Z Generating XML reports...
2022-09-27T16:45:15.8574839Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220927164456.xml
2022-09-27T16:45:16.3033327Z Running distributed/test_c10d_object_collectives ... [2022-09-27 16:45:16.302737]
2022-09-27T16:45:16.3034424Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:45:16.302837]
2022-09-27T16:45:18.2557023Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2022-09-27T16:45:18.2574806Z 
2022-09-27T16:45:18.2575271Z Running tests...
2022-09-27T16:45:18.2575797Z ----------------------------------------------------------------------
2022-09-27T16:45:19.8633856Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:19.8831813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59471
2022-09-27T16:45:19.8838137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59472
2022-09-27T16:45:21.5745340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:21.5745833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:21.5748000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:21.5748543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:21.5846976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:21.5847485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:21.5852965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:21.5853527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:21.8546714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:21.8551423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:21.8567405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:21.8573833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:21.8574859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:21.8655189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:24.0605902Z ok (5.803s)
2022-09-27T16:45:24.0625727Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59552
2022-09-27T16:45:24.0632437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59553
2022-09-27T16:45:25.7648450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.7649032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.7650230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.7650791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.7851860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.7852355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.7856846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.7857386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:26.0601892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:26.0602446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:26.0606775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:26.0607299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:26.0608111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:26.0608828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:28.1726580Z ok (4.112s)
2022-09-27T16:45:28.1749743Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59633
2022-09-27T16:45:28.1757135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59634
2022-09-27T16:45:29.8363303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:29.8363879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:29.8365034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:29.8365548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:29.8696202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:29.8696773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:29.8700762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:29.8701320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:30.1154673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:30.1159590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:30.1350441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:30.1356437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:30.1357482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:30.1363554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:32.2853752Z ok (4.113s)
2022-09-27T16:45:32.2875953Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59716
2022-09-27T16:45:32.2882711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59717
2022-09-27T16:45:33.9932521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:33.9933196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:33.9933818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:33.9934369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:34.0063598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:34.0064064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:34.0068393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:34.0068886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:34.2853071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:34.2859472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:34.2872837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:34.2878449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:34.2879367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:34.2962387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:36.4000912Z ok (4.115s)
2022-09-27T16:45:36.4001154Z 
2022-09-27T16:45:36.4001599Z ----------------------------------------------------------------------
2022-09-27T16:45:36.4001954Z Ran 4 tests in 18.142s
2022-09-27T16:45:36.4002819Z 
2022-09-27T16:45:36.4002950Z OK
2022-09-27T16:45:36.4003073Z 
2022-09-27T16:45:36.4003222Z Generating XML reports...
2022-09-27T16:45:36.4046686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220927164518.xml
2022-09-27T16:45:36.8668131Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-09-27 16:45:36.866232]
2022-09-27T16:45:36.8669032Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:45:36.866331]
2022-09-27T16:45:38.7816215Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-09-27T16:45:38.7832206Z 
2022-09-27T16:45:38.7832494Z Running tests...
2022-09-27T16:45:38.7832962Z ----------------------------------------------------------------------
2022-09-27T16:45:40.3724796Z   test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:40.4583330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59834
2022-09-27T16:45:40.4588400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59835
2022-09-27T16:45:40.4595647Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59836
2022-09-27T16:45:40.4601834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59837
2022-09-27T16:45:42.1593295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:42.1593824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:42.1594407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:42.1595136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.1596520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:42.1597018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:42.1600286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:42.1600763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.1893750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:42.1894223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:42.1897563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:42.1898038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.2359221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:42.2359712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:42.2360635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:42.2361121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.4243844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:42.4310822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:42.4432110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:45:42.4832075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:45:43.0673955Z skip: Need at least 4 CUDA devices (4.284s)
2022-09-27T16:45:43.0696553Z   test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59970
2022-09-27T16:45:43.0703408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59971
2022-09-27T16:45:43.0710451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59972
2022-09-27T16:45:43.0717710Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59973
2022-09-27T16:45:44.7625660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:44.7626195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:44.7626784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:44.7627266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:44.7718828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:44.7719226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:44.7722432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:44.7722881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:44.7874815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:44.7875200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:44.7878620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:44.7879107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:44.8433771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:44.8434258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:44.8434896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:44.8435359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:45.0374994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:45.0386561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:45.0503010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:45:45.0953424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:45:45.6788051Z skip: Need at least 4 CUDA devices (2.611s)
2022-09-27T16:45:45.6812036Z   test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60106
2022-09-27T16:45:45.6819238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60107
2022-09-27T16:45:45.6826593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60108
2022-09-27T16:45:45.6833532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60109
2022-09-27T16:45:47.3936231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.3936739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.3937369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.3937823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:47.3938423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.3938876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.3940388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.3940942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:47.4110612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.4111076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.4114157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.4114655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:47.4638225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.4638723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.4639576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.4640090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:47.6861252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:45:47.6862100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:47.6886461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:47.7139366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:45:48.2903686Z skip: Need at least 4 CUDA devices (2.611s)
2022-09-27T16:45:48.2928505Z   test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60242
2022-09-27T16:45:48.2935782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60243
2022-09-27T16:45:48.2942100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60244
2022-09-27T16:45:48.2949518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60245
2022-09-27T16:45:50.0119413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:50.0119952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:50.0120577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:50.0121064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:50.0128568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:50.0129016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:50.0132370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:50.0133095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:50.0229993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:50.0230445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:50.0233095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:50.0233595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:50.0242901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:50.0243374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:50.0246056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:50.0246552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:50.2770053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:50.2904394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:50.2937705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:45:50.2938288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:45:50.8018211Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:45:50.8041892Z   test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60378
2022-09-27T16:45:50.8048963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60379
2022-09-27T16:45:50.8057024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60380
2022-09-27T16:45:50.8064312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60381
2022-09-27T16:45:52.4810087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.4811369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.4812541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.4813392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.4814560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.4815420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.4816623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.4817589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.5166426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.5167412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.5169026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.5170235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.5640077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.5640935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.5641829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.5642591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.7568923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:45:52.7647434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:52.7735533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:45:52.8078549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:53.4134464Z skip: Need at least 4 CUDA devices (2.611s)
2022-09-27T16:45:53.4134955Z 
2022-09-27T16:45:53.4135565Z ----------------------------------------------------------------------
2022-09-27T16:45:53.4136180Z Ran 5 tests in 14.630s
2022-09-27T16:45:53.4136433Z 
2022-09-27T16:45:53.4136602Z OK (skipped=5)
2022-09-27T16:45:53.4136850Z 
2022-09-27T16:45:53.4137062Z Generating XML reports...
2022-09-27T16:45:53.4182927Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220927164538.xml
2022-09-27T16:45:53.8645871Z Running distributed/fsdp/test_fsdp_overlap ... [2022-09-27 16:45:53.863992]
2022-09-27T16:45:53.8646649Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:45:53.864092]
2022-09-27T16:45:55.7625450Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-09-27T16:45:55.7641948Z 
2022-09-27T16:45:55.7642445Z Running tests...
2022-09-27T16:45:55.7643153Z ----------------------------------------------------------------------
2022-09-27T16:45:57.3927966Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:57.4118494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60549
2022-09-27T16:45:59.0528543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:59.0529390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:59.0530516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:59.0531273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:59.2948935Z dist init r=0, world=1
2022-09-27T16:45:59.2953423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:59.2954404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-09-27T16:46:00.6772719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:00.7711880Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:46:00.7712753Z   warnings.warn(
2022-09-27T16:46:09.1085357Z 
2022-09-27T16:46:09.1085820Z rank0:
2022-09-27T16:46:09.1086616Z   e1: {'cpu_iter': 0.0018595897000002637, 'cpu_wait': 3.408999999994222e-05, 'gpu_compute': 0.05031359987333417, 'gpu_total': 0.693699198961258}
2022-09-27T16:46:09.1087209Z   e2: {'cpu_iter': 0.004760296700000044, 'cpu_wait': 3.358019999986084e-05, 'gpu_compute': 0.21424960158765316, 'gpu_total': 1.9864128232002258}
2022-09-27T16:46:09.1087808Z   e3: {'cpu_iter': 0.002100144699999884, 'cpu_wait': 0.1836176424999998, 'gpu_compute': 186.1791477203369, 'gpu_total': 186.46436614990233}
2022-09-27T16:46:09.1088398Z   e4: {'cpu_iter': 0.004822106500000168, 'cpu_wait': 0.18191452759999988, 'gpu_compute': 186.28162803649903, 'gpu_total': 186.7400680541992}
2022-09-27T16:46:09.5322291Z ok (13.768s)
2022-09-27T16:46:09.5330254Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-09-27T16:46:09.5330997Z 
2022-09-27T16:46:09.5331312Z ----------------------------------------------------------------------
2022-09-27T16:46:09.5331684Z Ran 2 tests in 13.769s
2022-09-27T16:46:09.5331866Z 
2022-09-27T16:46:09.5331959Z OK (skipped=1)
2022-09-27T16:46:09.5332129Z 
2022-09-27T16:46:09.5332265Z Generating XML reports...
2022-09-27T16:46:09.5382929Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220927164555.xml
2022-09-27T16:46:09.5386343Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220927164555.xml
2022-09-27T16:46:09.9773058Z Running distributed/_shard/checkpoint/test_file_system_checkpoint_cpu ... [2022-09-27 16:46:09.976763]
2022-09-27T16:46:09.9773898Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:46:09.976857]
2022-09-27T16:46:11.9128955Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu
2022-09-27T16:46:11.9154934Z 
2022-09-27T16:46:11.9155124Z Running tests...
2022-09-27T16:46:11.9155624Z ----------------------------------------------------------------------
2022-09-27T16:46:13.5762551Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:13.5960154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60627
2022-09-27T16:46:13.5966453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60628
2022-09-27T16:46:15.3052590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:15.3053385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:15.3056415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:15.3056899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:15.3219763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:15.3220211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:15.3224671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:15.3225170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:15.5792575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:15.5886005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:15.6099334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:15.6099851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:15.6100633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:15.6101317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:16.1025449Z ok (4.187s)
2022-09-27T16:46:16.1066017Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60701
2022-09-27T16:46:16.1072610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60702
2022-09-27T16:46:17.7697500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:17.7698069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:17.7700267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:17.7700773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:17.8175643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:17.8176118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:17.8179732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:17.8180243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:18.0316742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:18.0695908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:18.0830589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:18.0831119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:18.0831921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:18.0832601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:18.7135521Z ok (2.611s)
2022-09-27T16:46:18.7159249Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60775
2022-09-27T16:46:18.7167510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60776
2022-09-27T16:46:20.3807997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:20.3808519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:20.3810712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:20.3811193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:20.4287779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:20.4288258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:20.4291784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:20.4292261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:20.6421311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:20.6799113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:20.7012110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:20.7012917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:20.7013909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:20.7014615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:21.1223932Z ok (2.409s)
2022-09-27T16:46:21.1256806Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60849
2022-09-27T16:46:21.1263112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60850
2022-09-27T16:46:22.7993026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:22.7993764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:22.7994751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:22.7995327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:22.8094130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:22.8094582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:22.8098472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:22.8098977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:23.0692158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:23.0692719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:23.0905576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:23.0906106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:23.0906951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:23.0907645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:23.8324904Z ok (2.710s)
2022-09-27T16:46:23.8538481Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.021s)
2022-09-27T16:46:23.8565313Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60923
2022-09-27T16:46:23.8572168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60924
2022-09-27T16:46:25.5134668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:25.5135166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:25.5135986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:25.5136457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:25.5137814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:25.5138346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:25.5139000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:25.5139484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:25.7876256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:25.7876862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:25.7989639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:25.7990221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:25.7991288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:25.7992022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:26.2628713Z ok (2.409s)
2022-09-27T16:46:26.2628956Z 
2022-09-27T16:46:26.2629411Z ----------------------------------------------------------------------
2022-09-27T16:46:26.2629766Z Ran 6 tests in 14.347s
2022-09-27T16:46:26.2629941Z 
2022-09-27T16:46:26.2630038Z OK
2022-09-27T16:46:26.2630177Z 
2022-09-27T16:46:26.2630293Z Generating XML reports...
2022-09-27T16:46:26.2673621Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220927164611.xml
2022-09-27T16:46:26.2676946Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220927164611.xml
2022-09-27T16:46:26.2680825Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927164611.xml
2022-09-27T16:46:26.7288804Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-09-27 16:46:26.728342]
2022-09-27T16:46:26.7289917Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:46:26.728434]
2022-09-27T16:46:28.6857736Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-09-27T16:46:28.6875571Z 
2022-09-27T16:46:28.6875830Z Running tests...
2022-09-27T16:46:28.6876322Z ----------------------------------------------------------------------
2022-09-27T16:46:30.2503410Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:30.3313308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61032
2022-09-27T16:46:30.3318280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61033
2022-09-27T16:46:30.3325629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61034
2022-09-27T16:46:30.3333247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61035
2022-09-27T16:46:32.0114289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.0115015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.0115634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.0116117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.0391664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.0392152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.0395314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.0395808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.0418873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.0419353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.0421738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.0422218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.0585239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.0585964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.0588753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.0589242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.2951334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:46:32.3062256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:32.3069350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:46:32.3074191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:32.8404170Z skip: Need at least 4 CUDA devices (4.152s)
2022-09-27T16:46:32.8428381Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61168
2022-09-27T16:46:32.8435129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61169
2022-09-27T16:46:32.8442108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61170
2022-09-27T16:46:32.8449183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61171
2022-09-27T16:46:34.5327850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:34.5328424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:34.5329027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:34.5329469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:34.5335517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:34.5336005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:34.5339093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:34.5339839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:34.5511145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:34.5511598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:34.5514103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:34.5514534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:34.5764989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:34.5765428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:34.5767366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:34.5767865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:34.7989626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:46:34.8161727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:34.8257161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:46:34.8266103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:35.3517683Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:46:35.3543902Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61304
2022-09-27T16:46:35.3551567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61305
2022-09-27T16:46:35.3558582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61306
2022-09-27T16:46:35.3565822Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61307
2022-09-27T16:46:37.0252578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.0253159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.0253771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.0254220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.0600415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.0601168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.0601796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.0602236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.0603125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.0603577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.0604186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.0604648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.0835632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.0836145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.0838754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.0839248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.3117561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:46:37.3284475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:37.3336880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:46:37.3351067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:37.8633423Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:46:37.8656872Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61440
2022-09-27T16:46:37.8663400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61441
2022-09-27T16:46:37.8669899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61442
2022-09-27T16:46:37.8676500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61443
2022-09-27T16:46:39.5632127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:39.5632683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:39.5633257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:39.5633751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:39.5657712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:39.5658192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:39.5660369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:39.5661001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:39.5736732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:39.5737199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:39.5739701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:39.5740175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:39.5850265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:39.5850931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:39.5854306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:39.5854795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:39.8311290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:39.8385665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:46:39.8386218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:39.8494767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:46:40.3745177Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:46:40.3745480Z 
2022-09-27T16:46:40.3745912Z ----------------------------------------------------------------------
2022-09-27T16:46:40.3746226Z Ran 4 tests in 11.687s
2022-09-27T16:46:40.3746399Z 
2022-09-27T16:46:40.3746513Z OK (skipped=4)
2022-09-27T16:46:40.3750318Z 
2022-09-27T16:46:40.3750695Z Generating XML reports...
2022-09-27T16:46:40.3790401Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220927164628.xml
2022-09-27T16:46:40.8345380Z Running distributed/elastic/timer/local_timer_example ... [2022-09-27 16:46:40.833952]
2022-09-27T16:46:40.8346181Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:46:40.834044]
2022-09-27T16:46:42.7494796Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-09-27T16:46:42.7510162Z 
2022-09-27T16:46:42.7510451Z Running tests...
2022-09-27T16:46:42.7510898Z ----------------------------------------------------------------------
2022-09-27T16:46:44.3661114Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-09-27 16:46:44,365 driver: init
2022-09-27T16:46:44.4497435Z [INFO] 2022-09-27 16:46:44,449 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-09-27T16:46:44.4498001Z [INFO] 2022-09-27 16:46:44,449 api: Starting watchdog thread...
2022-09-27T16:46:46.1611415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1611972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1613155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1613617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1653064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1653533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1658030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1658538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1661781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1662252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1666791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1667295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1673394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1673872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1677848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1678315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1700794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1701274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1705480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1705937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1716467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1716931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1721084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1721577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1755676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1756343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1760644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1761079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.1829741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:46.1830209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:46.1834524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:46.1834977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:46.4583007Z [INFO] 2022-09-27 16:46:46,457 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4653579Z [INFO] 2022-09-27 16:46:46,464 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4702322Z [INFO] 2022-09-27 16:46:46,469 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4804092Z [INFO] 2022-09-27 16:46:46,480 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4806549Z [INFO] 2022-09-27 16:46:46,480 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4820111Z [INFO] 2022-09-27 16:46:46,481 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4835800Z [INFO] 2022-09-27 16:46:46,483 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:46.4910759Z [INFO] 2022-09-27 16:46:46,490 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:47.5726884Z [INFO] 2022-09-27 16:46:47,571 api: Reaping worker_id=[61616]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:47.5728355Z [INFO] 2022-09-27 16:46:47,572 api: Successfully reaped worker=[61616]
2022-09-27T16:46:47.5830414Z [INFO] 2022-09-27 16:46:47,582 api: Reaping worker_id=[61618]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:47.5834018Z [INFO] 2022-09-27 16:46:47,583 api: Successfully reaped worker=[61618]
2022-09-27T16:46:47.5939979Z [INFO] 2022-09-27 16:46:47,593 api: Reaping worker_id=[61614]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:47.5943334Z [INFO] 2022-09-27 16:46:47,593 api: Successfully reaped worker=[61614]
2022-09-27T16:46:47.6046697Z [INFO] 2022-09-27 16:46:47,604 api: Reaping worker_id=[61612]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:47.6052248Z [INFO] 2022-09-27 16:46:47,604 api: Successfully reaped worker=[61612]
2022-09-27T16:46:47.6160543Z [INFO] 2022-09-27 16:46:47,615 api: Stopping LocalTimerServer
2022-09-27T16:46:47.6160984Z [INFO] 2022-09-27 16:46:47,615 api: Stopping watchdog thread...
2022-09-27T16:46:47.6259608Z ok (4.875s)
2022-09-27T16:46:47.6279802Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-09-27 16:46:47,627 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-09-27T16:46:47.6280346Z [INFO] 2022-09-27 16:46:47,627 api: Starting watchdog thread...
2022-09-27T16:46:49.3647846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.3648987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.3650641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.3651564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.3652754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.3653713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.3654925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.3656375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.3659846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.3660797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.3664209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.3665193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.3740442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.3741399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.3744626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.3745548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.3804291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.3805254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.3808878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.3810253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.4096406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.4097422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.4099059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.4100026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.4137357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.4138268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.4141491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.4142501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.4332276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:49.4333199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:49.4338076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:49.4339071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:49.6389395Z [INFO] 2022-09-27 16:46:49,638 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6475820Z [INFO] 2022-09-27 16:46:49,647 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6614354Z [INFO] 2022-09-27 16:46:49,660 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6662222Z [INFO] 2022-09-27 16:46:49,665 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6691300Z [INFO] 2022-09-27 16:46:49,668 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6811926Z [INFO] 2022-09-27 16:46:49,680 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6850855Z [INFO] 2022-09-27 16:46:49,684 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:49.6995967Z [INFO] 2022-09-27 16:46:49,699 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.5896247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.5897155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.5897802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.5898235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.5898812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.5899280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.5899870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.5900336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.5900938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.5901416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.5902026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.5902479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.5949506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.5949980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.5953919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.5954389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.6616472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.6617279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.6618264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.6618798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.6726518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.6726999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.6730398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.6730920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.6731533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.6731975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.6736632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.6737148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.7325763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:52.7326654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:52.7327677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:52.7328177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:52.8658805Z [INFO] 2022-09-27 16:46:52,865 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.8916378Z [INFO] 2022-09-27 16:46:52,891 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.8918164Z [INFO] 2022-09-27 16:46:52,891 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.8954728Z [INFO] 2022-09-27 16:46:52,895 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.9331593Z [INFO] 2022-09-27 16:46:52,932 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.9569693Z [INFO] 2022-09-27 16:46:52,956 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.9592167Z [INFO] 2022-09-27 16:46:52,958 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:52.9892193Z [INFO] 2022-09-27 16:46:52,988 api: Timer client configured to: LocalTimerClient
2022-09-27T16:46:53.9641764Z [INFO] 2022-09-27 16:46:53,963 api: Reaping worker_id=[62157]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:53.9642659Z [INFO] 2022-09-27 16:46:53,963 api: Successfully reaped worker=[62157]
2022-09-27T16:46:53.9950646Z [INFO] 2022-09-27 16:46:53,994 api: Reaping worker_id=[62164]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:53.9952854Z [INFO] 2022-09-27 16:46:53,994 api: Successfully reaped worker=[62164]
2022-09-27T16:46:53.9953947Z [INFO] 2022-09-27 16:46:53,995 api: Reaping worker_id=[62161]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:53.9956808Z [INFO] 2022-09-27 16:46:53,995 api: Successfully reaped worker=[62161]
2022-09-27T16:46:53.9958074Z [INFO] 2022-09-27 16:46:53,995 api: Reaping worker_id=[62158]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:53.9958877Z [INFO] 2022-09-27 16:46:53,995 local_timer: Process with pid=62158 does not exist. Skipping
2022-09-27T16:46:53.9959346Z [INFO] 2022-09-27 16:46:53,995 api: Successfully reaped worker=[62158]
2022-09-27T16:46:54.0285656Z [INFO] 2022-09-27 16:46:54,028 api: Stopping LocalTimerServer
2022-09-27T16:46:54.0286414Z [INFO] 2022-09-27 16:46:54,028 api: Stopping watchdog thread...
2022-09-27T16:46:54.0366660Z [INFO] 2022-09-27 16:46:54,036 api: Reaping worker_id=[62159]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-09-27T16:46:54.0367535Z [INFO] 2022-09-27 16:46:54,036 local_timer: Process with pid=62159 does not exist. Skipping
2022-09-27T16:46:54.0368032Z [INFO] 2022-09-27 16:46:54,036 api: Successfully reaped worker=[62159]
2022-09-27T16:46:54.0371426Z ok (6.411s)
2022-09-27T16:46:54.0373573Z 
2022-09-27T16:46:54.0374123Z ----------------------------------------------------------------------
2022-09-27T16:46:54.0374924Z Ran 2 tests in 11.286s
2022-09-27T16:46:54.0375195Z 
2022-09-27T16:46:54.0375304Z OK
2022-09-27T16:46:54.0375448Z 
2022-09-27T16:46:54.0375579Z Generating XML reports...
2022-09-27T16:46:54.0416398Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220927164642.xml
2022-09-27T16:46:54.4830415Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-09-27 16:46:54.482393]
2022-09-27T16:46:54.4832033Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:46:54.482492]
2022-09-27T16:46:56.4394979Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-09-27T16:46:56.4411683Z 
2022-09-27T16:46:56.4412085Z Running tests...
2022-09-27T16:46:56.4412632Z ----------------------------------------------------------------------
2022-09-27T16:46:58.1060765Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:58.1246937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62464
2022-09-27T16:46:58.1254330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62465
2022-09-27T16:46:59.8270549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:59.8271054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:59.8272865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:59.8273377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:59.8408048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:59.8408500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:59.8412780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:59.8413463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:00.0846642Z dist init r=1, world=2
2022-09-27T16:47:00.0851023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:00.0872722Z dist init r=0, world=2
2022-09-27T16:47:00.0878640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:00.0879473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:00.0953646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:01.6137350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:01.6138332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:02.3354058Z ok (5.894s)
2022-09-27T16:47:02.3383671Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62549
2022-09-27T16:47:02.3389436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62550
2022-09-27T16:47:04.0147038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:04.0147610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:04.0148778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:04.0149277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:04.0503968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:04.0504475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:04.0506873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:04.0507391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:04.2747724Z dist init r=1, world=2
2022-09-27T16:47:04.2751927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:04.2944005Z dist init r=0, world=2
2022-09-27T16:47:04.2949586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:04.2950854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:04.2956164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:05.8287947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:05.8288517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:06.5487971Z ok (4.213s)
2022-09-27T16:47:06.5488306Z 
2022-09-27T16:47:06.5488753Z ----------------------------------------------------------------------
2022-09-27T16:47:06.5489132Z Ran 2 tests in 10.107s
2022-09-27T16:47:06.5489875Z 
2022-09-27T16:47:06.5489964Z OK
2022-09-27T16:47:06.5490116Z 
2022-09-27T16:47:06.5490263Z Generating XML reports...
2022-09-27T16:47:06.5540692Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220927164656.xml
2022-09-27T16:47:06.9846910Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-09-27 16:47:06.984151]
2022-09-27T16:47:06.9847758Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:06.984248]
2022-09-27T16:47:08.9167454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzndlas6r
2022-09-27T16:47:08.9168135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzndlas6r/_remote_module_non_scriptable.py
2022-09-27T16:47:08.9569324Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-09-27T16:47:08.9590692Z 
2022-09-27T16:47:08.9590845Z Running tests...
2022-09-27T16:47:08.9591285Z ----------------------------------------------------------------------
2022-09-27T16:47:10.6092456Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:10.6294541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62669
2022-09-27T16:47:10.6300905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62670
2022-09-27T16:47:10.6308251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62671
2022-09-27T16:47:10.6315118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62672
2022-09-27T16:47:12.2931788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:12.2932614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:12.2933305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:12.2933931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:12.2940475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:12.2941020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:12.2945160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:12.2945854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:12.2990509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:12.2991184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:12.2994977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:12.2995670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:12.3142272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:12.3142941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:12.3146311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:12.5697603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:12.5698317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rzltka0
2022-09-27T16:47:12.5698933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rzltka0/_remote_module_non_scriptable.py
2022-09-27T16:47:12.5701503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaathc_o3
2022-09-27T16:47:12.5704783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaathc_o3/_remote_module_non_scriptable.py
2022-09-27T16:47:12.5772985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i4_a0fd
2022-09-27T16:47:12.5775842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i4_a0fd/_remote_module_non_scriptable.py
2022-09-27T16:47:12.5850618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpep6u80eu
2022-09-27T16:47:12.5854493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpep6u80eu/_remote_module_non_scriptable.py
2022-09-27T16:47:12.6078508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:12.6081087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:12.6142529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:12.6228301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:13.1386499Z ok (4.179s)
2022-09-27T16:47:13.1392364Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-09-27T16:47:13.1409118Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62805
2022-09-27T16:47:13.1416809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62806
2022-09-27T16:47:13.1424494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62807
2022-09-27T16:47:13.1432016Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62808
2022-09-27T16:47:14.8305357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:14.8306236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:14.8307178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:14.8307622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:14.8321166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:14.8321556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:14.8325857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:14.8326485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:14.8486217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:14.8486638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:14.8491267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:14.8491724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:14.8690770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:14.8691212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:14.8694947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:14.8695445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:15.1100897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv12nuflz
2022-09-27T16:47:15.1102239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv12nuflz/_remote_module_non_scriptable.py
2022-09-27T16:47:15.1258153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvg4lnrju
2022-09-27T16:47:15.1261360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvg4lnrju/_remote_module_non_scriptable.py
2022-09-27T16:47:15.1287618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm9ul49c
2022-09-27T16:47:15.1290251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm9ul49c/_remote_module_non_scriptable.py
2022-09-27T16:47:15.1322186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5z5nvej
2022-09-27T16:47:15.1325515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5z5nvej/_remote_module_non_scriptable.py
2022-09-27T16:47:15.1492606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:15.1638134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:15.1660478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:15.1701208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:15.6499894Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:47:15.6507078Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-09-27T16:47:15.6523015Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62941
2022-09-27T16:47:15.6529622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62942
2022-09-27T16:47:15.6537425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62943
2022-09-27T16:47:15.6544296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62944
2022-09-27T16:47:17.3132174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:17.3132686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:17.3133701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:17.3134153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:17.3137953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:17.3138359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:17.3142501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:17.3142943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:17.3452472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:17.3452881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:17.3457462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:17.3457975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:17.3527151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:17.3527575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:17.3531173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:17.3531643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:17.5835766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajjo8b1r
2022-09-27T16:47:17.5836682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajjo8b1r/_remote_module_non_scriptable.py
2022-09-27T16:47:17.5997056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy5algl9
2022-09-27T16:47:17.5999914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy5algl9/_remote_module_non_scriptable.py
2022-09-27T16:47:17.6066512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbs3nz65h
2022-09-27T16:47:17.6069235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbs3nz65h/_remote_module_non_scriptable.py
2022-09-27T16:47:17.6112946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv81swuw
2022-09-27T16:47:17.6116476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv81swuw/_remote_module_non_scriptable.py
2022-09-27T16:47:17.6201396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:17.6368478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:17.6442510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:17.6501173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:18.1613259Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:47:18.1740024Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.013s)
2022-09-27T16:47:18.1812499Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.007s)
2022-09-27T16:47:18.1889238Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.008s)
2022-09-27T16:47:18.1908836Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-09-27T16:47:18.1921311Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-09-27T16:47:18.1934238Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-09-27T16:47:18.2034726Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s)
2022-09-27T16:47:18.2035263Z 
2022-09-27T16:47:18.2035938Z ----------------------------------------------------------------------
2022-09-27T16:47:18.2036534Z Ran 10 tests in 9.244s
2022-09-27T16:47:18.2036826Z 
2022-09-27T16:47:18.2037028Z OK (skipped=2)
2022-09-27T16:47:18.2037325Z 
2022-09-27T16:47:18.2037541Z Generating XML reports...
2022-09-27T16:47:18.2078248Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220927164708.xml
2022-09-27T16:47:18.2086885Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220927164708.xml
2022-09-27T16:47:18.6363124Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-09-27 16:47:18.635796]
2022-09-27T16:47:18.6363927Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:18.635891]
2022-09-27T16:47:20.5437032Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-09-27T16:47:20.5455015Z 
2022-09-27T16:47:20.5455278Z Running tests...
2022-09-27T16:47:20.5455732Z ----------------------------------------------------------------------
2022-09-27T16:47:22.1443995Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:22.2259964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63112
2022-09-27T16:47:22.2265141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63113
2022-09-27T16:47:22.2272777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63114
2022-09-27T16:47:22.2279508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63115
2022-09-27T16:47:23.9010970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.9011525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9012481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9013002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.9032375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.9032843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9036189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9036691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.9042427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.9042909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9045285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9045784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.9249026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.9249499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9252961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9253447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:24.1730214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:24.1755374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:24.1778879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:24.1834632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:24.7350685Z skip: Need at least 4 CUDA devices (4.189s)
2022-09-27T16:47:24.7375311Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63248
2022-09-27T16:47:24.7382486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63249
2022-09-27T16:47:24.7389692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63250
2022-09-27T16:47:24.7396827Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63251
2022-09-27T16:47:26.4332135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:26.4332689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:26.4333290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:26.4333803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:26.4537167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:26.4537677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:26.4538905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:26.4539403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:26.4589471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:26.4589963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:26.4593575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:26.4594391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:26.5134953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:26.5135461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:26.5136026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:26.5136514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:26.7159353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:26.7170312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:26.7294618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:26.7482181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:27.2464332Z skip: Need at least 4 CUDA devices (2.511s)
2022-09-27T16:47:27.2464597Z 
2022-09-27T16:47:27.2464995Z ----------------------------------------------------------------------
2022-09-27T16:47:27.2465376Z Ran 2 tests in 6.701s
2022-09-27T16:47:27.2465529Z 
2022-09-27T16:47:27.2465610Z OK (skipped=2)
2022-09-27T16:47:27.2465772Z 
2022-09-27T16:47:27.2465904Z Generating XML reports...
2022-09-27T16:47:27.2506553Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220927164720.xml
2022-09-27T16:47:27.6826396Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-09-27 16:47:27.682050]
2022-09-27T16:47:27.6827496Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:27.682149]
2022-09-27T16:47:29.5651002Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-09-27T16:47:29.5667306Z 
2022-09-27T16:47:29.5667780Z Running tests...
2022-09-27T16:47:29.5668278Z ----------------------------------------------------------------------
2022-09-27T16:47:31.1300925Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:31.2097888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63419
2022-09-27T16:47:31.2102349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63420
2022-09-27T16:47:31.2108693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63421
2022-09-27T16:47:31.2115330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63422
2022-09-27T16:47:32.9190309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:32.9190893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:32.9191464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:32.9191952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:32.9481345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:32.9481813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:32.9484632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:32.9485126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:32.9598664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:32.9599143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:32.9602614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:32.9603104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:32.9641321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:32.9644545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:32.9645139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:32.9645592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:33.2034157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:33.2097266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:33.2168186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:33.2199338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:33.8189078Z skip: Need at least 4 CUDA devices (4.252s)
2022-09-27T16:47:33.8217821Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63555
2022-09-27T16:47:33.8224187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63556
2022-09-27T16:47:33.8231088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63557
2022-09-27T16:47:33.8237735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63558
2022-09-27T16:47:35.5121761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.5122369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.5123267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.5123732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.5256373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.5256851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.5259407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.5260087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.5276434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.5276929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.5278250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.5278961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.5349720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.5350200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.5352712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.5353179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.7969154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:35.7990683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:35.8001091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:35.8024487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:36.3313666Z skip: Need at least 4 CUDA devices (2.512s)
2022-09-27T16:47:36.3313963Z 
2022-09-27T16:47:36.3314395Z ----------------------------------------------------------------------
2022-09-27T16:47:36.3314749Z Ran 2 tests in 6.765s
2022-09-27T16:47:36.3314922Z 
2022-09-27T16:47:36.3315042Z OK (skipped=2)
2022-09-27T16:47:36.3315185Z 
2022-09-27T16:47:36.3315317Z Generating XML reports...
2022-09-27T16:47:36.3355304Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220927164729.xml
2022-09-27T16:47:36.7583010Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-09-27 16:47:36.757768]
2022-09-27T16:47:36.7583833Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:36.757863]
2022-09-27T16:47:38.6256615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponxh_nvb
2022-09-27T16:47:38.6257840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponxh_nvb/_remote_module_non_scriptable.py
2022-09-27T16:47:38.6432124Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2022-09-27T16:47:38.6452791Z 
2022-09-27T16:47:38.6453254Z Running tests...
2022-09-27T16:47:38.6453767Z ----------------------------------------------------------------------
2022-09-27T16:47:40.3182760Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:40.3382563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63726
2022-09-27T16:47:40.3388996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63727
2022-09-27T16:47:40.3396201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63728
2022-09-27T16:47:40.3403679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63729
2022-09-27T16:47:41.9827844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:41.9828936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:41.9830122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:41.9831017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:41.9991035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:41.9992001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:41.9996051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:41.9997046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:42.0639766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:42.0640830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:42.0642437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:42.0643608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:42.1138662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:42.1139617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:42.1140676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:42.1141979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:42.2596717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qvx6lvl
2022-09-27T16:47:42.2597862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qvx6lvl/_remote_module_non_scriptable.py
2022-09-27T16:47:42.2628104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6woqntno
2022-09-27T16:47:42.2630573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6woqntno/_remote_module_non_scriptable.py
2022-09-27T16:47:42.2738039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:42.2774381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:42.3106142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuffji9v
2022-09-27T16:47:42.3107140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuffji9v/_remote_module_non_scriptable.py
2022-09-27T16:47:42.3256329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:42.3681573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhylg7sv
2022-09-27T16:47:42.3682756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhylg7sv/_remote_module_non_scriptable.py
2022-09-27T16:47:42.3841592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:42.9478562Z skip: Need at least 4 CUDA devices (4.302s)
2022-09-27T16:47:42.9498369Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63862
2022-09-27T16:47:42.9505154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63863
2022-09-27T16:47:42.9511542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63864
2022-09-27T16:47:42.9518660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63865
2022-09-27T16:47:44.7357975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:44.7358958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:44.7360133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:44.7361042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:44.7403547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:44.7404461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:44.7408248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:44.7409179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:44.7661413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:44.7662305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:44.7664917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:44.7665827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:44.7878951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:44.7879909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:44.7882376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:44.7883319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:45.0122790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jc6hi89
2022-09-27T16:47:45.0123999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jc6hi89/_remote_module_non_scriptable.py
2022-09-27T16:47:45.0264510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:47:45.0273219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw2wxb28
2022-09-27T16:47:45.0275486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw2wxb28/_remote_module_non_scriptable.py
2022-09-27T16:47:45.0284449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t_gm_i1
2022-09-27T16:47:45.0286871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t_gm_i1/_remote_module_non_scriptable.py
2022-09-27T16:47:45.0415192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:45.0430990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:47:45.0490868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpium9kr3l
2022-09-27T16:47:45.0493062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpium9kr3l/_remote_module_non_scriptable.py
2022-09-27T16:47:45.0645641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:45.6593564Z skip: Need at least 4 CUDA devices (2.711s)
2022-09-27T16:47:45.6593847Z 
2022-09-27T16:47:45.6594294Z ----------------------------------------------------------------------
2022-09-27T16:47:45.6594629Z Ran 2 tests in 7.014s
2022-09-27T16:47:45.6594808Z 
2022-09-27T16:47:45.6594923Z OK (skipped=2)
2022-09-27T16:47:45.6595086Z 
2022-09-27T16:47:45.6595530Z Generating XML reports...
2022-09-27T16:47:45.6637563Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220927164738.xml
2022-09-27T16:47:46.1133504Z Running distributed/fsdp/test_fsdp_uneven ... [2022-09-27 16:47:46.112849]
2022-09-27T16:47:46.1134247Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:46.112948]
2022-09-27T16:47:47.9985103Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-09-27T16:47:47.9999708Z 
2022-09-27T16:47:48.0000194Z Running tests...
2022-09-27T16:47:48.0000660Z ----------------------------------------------------------------------
2022-09-27T16:47:48.0010101Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-09-27T16:47:49.6305008Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:49.6493804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64033
2022-09-27T16:47:49.6500082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64034
2022-09-27T16:47:51.3024901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:51.3025443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:51.3026322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:51.3026840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:51.3339217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:51.3339959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:51.3342873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:51.3343647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:51.5743528Z dist init r=0, world=2
2022-09-27T16:47:51.5747555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:51.5977474Z dist init r=1, world=2
2022-09-27T16:47:51.5983559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:51.5984847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:51.6053294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:53.1259562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:53.1260167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:54.1606153Z ok (6.160s)
2022-09-27T16:47:54.1606426Z 
2022-09-27T16:47:54.1606910Z ----------------------------------------------------------------------
2022-09-27T16:47:54.1607223Z Ran 1 test in 6.160s
2022-09-27T16:47:54.1607409Z 
2022-09-27T16:47:54.1608302Z OK
2022-09-27T16:47:54.1608483Z 
2022-09-27T16:47:54.1608901Z Generating XML reports...
2022-09-27T16:47:54.1646547Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220927164747.xml
2022-09-27T16:47:54.6298598Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-09-27 16:47:54.629307]
2022-09-27T16:47:54.6299392Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:47:54.629407]
2022-09-27T16:47:56.5919848Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-09-27T16:47:56.5935368Z 
2022-09-27T16:47:56.5935532Z Running tests...
2022-09-27T16:47:56.5936500Z ----------------------------------------------------------------------
2022-09-27T16:47:56.5945518Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-09-27T16:47:58.2624174Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:58.2945107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64153
2022-09-27T16:47:58.2950557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64154
2022-09-27T16:47:59.9359586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:59.9360086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:59.9361113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:59.9361758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:59.9718965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:59.9719441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:59.9722619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:59.9723371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:00.2103489Z dist init r=0, world=2
2022-09-27T16:48:00.2108165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:00.2352775Z dist init r=1, world=2
2022-09-27T16:48:00.2358994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:00.2359886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:00.2414096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:01.7611830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:01.7612403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:01.7836852Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:48:01.7837969Z   warnings.warn(
2022-09-27T16:48:01.7839407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1414: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-09-27T16:48:01.7840421Z   warnings.warn(
2022-09-27T16:48:02.8057362Z ok (6.212s)
2022-09-27T16:48:02.8057663Z 
2022-09-27T16:48:02.8058107Z ----------------------------------------------------------------------
2022-09-27T16:48:02.8058471Z Ran 1 test in 6.212s
2022-09-27T16:48:02.8058644Z 
2022-09-27T16:48:02.8058749Z OK
2022-09-27T16:48:02.8058868Z 
2022-09-27T16:48:02.8059011Z Generating XML reports...
2022-09-27T16:48:02.8099698Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220927164756.xml
2022-09-27T16:48:03.2445296Z Running distributed/fsdp/test_fsdp_fx ... [2022-09-27 16:48:03.243998]
2022-09-27T16:48:03.2446122Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:03.244096]
2022-09-27T16:48:05.2008230Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2022-09-27T16:48:05.2024397Z 
2022-09-27T16:48:05.2024734Z Running tests...
2022-09-27T16:48:05.2025434Z ----------------------------------------------------------------------
2022-09-27T16:48:05.2046536Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2022-09-27T16:48:06.8949715Z test ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos`` ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:06.9147642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64273
2022-09-27T16:48:06.9154033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64274
2022-09-27T16:48:08.5779409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:08.5779954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:08.5780986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:08.5781477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:08.5988393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:08.5988882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:08.5993400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:08.5993873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:08.8507188Z dist init r=0, world=2
2022-09-27T16:48:08.8511500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:08.8619614Z dist init r=1, world=2
2022-09-27T16:48:08.8625598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:08.8626596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:08.8717357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:10.3717339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:10.3717953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:10.9247391Z ok (5.722s)
2022-09-27T16:48:10.9247654Z 
2022-09-27T16:48:10.9248104Z ----------------------------------------------------------------------
2022-09-27T16:48:10.9248471Z Ran 1 test in 5.722s
2022-09-27T16:48:10.9248637Z 
2022-09-27T16:48:10.9248715Z OK
2022-09-27T16:48:10.9248875Z 
2022-09-27T16:48:10.9249024Z Generating XML reports...
2022-09-27T16:48:10.9299995Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220927164805.xml
2022-09-27T16:48:11.3743895Z Running distributed/test_data_parallel ... [2022-09-27 16:48:11.373830]
2022-09-27T16:48:11.3744706Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:11.373927]
2022-09-27T16:48:14.9356794Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-09-27T16:48:15.0071459Z 
2022-09-27T16:48:15.0071944Z Running tests...
2022-09-27T16:48:15.0072585Z ----------------------------------------------------------------------
2022-09-27T16:48:16.9124736Z   test_autocast (__main__.TestDataParallel) ... ok (1.905s)
2022-09-27T16:48:17.1105807Z   test_data_parallel (__main__.TestDataParallel) ... ok (0.198s)
2022-09-27T16:48:17.1234495Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.013s)
2022-09-27T16:48:17.1264327Z   test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.003s)
2022-09-27T16:48:17.1324956Z   test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s)
2022-09-27T16:48:17.1382896Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s)
2022-09-27T16:48:17.1398291Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-09-27T16:48:17.1399077Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-09-27T16:48:17.1407467Z ok (0.002s)
2022-09-27T16:48:17.1441765Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-09-27T16:48:17.1749113Z Test device[0] check at forward time. ... ok (0.034s)
2022-09-27T16:48:17.3048901Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.130s)
2022-09-27T16:48:17.3100022Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s)
2022-09-27T16:48:17.3157284Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".
2022-09-27T16:48:17.3157898Z   warnings.warn(
2022-09-27T16:48:17.3315347Z ok (0.021s)
2022-09-27T16:48:17.3345537Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s)
2022-09-27T16:48:17.3403942Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s)
2022-09-27T16:48:17.3445177Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s)
2022-09-27T16:48:18.3484437Z   test_data_parallel_rnn (__main__.TestDataParallel) ... ok (1.004s)
2022-09-27T16:48:18.3520435Z   test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.004s)
2022-09-27T16:48:18.3640616Z   test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s)
2022-09-27T16:48:18.3861723Z   test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
2022-09-27T16:48:18.3862469Z   warnings.warn('Was asked to gather along dimension 0, but all '
2022-09-27T16:48:18.4070825Z ok (0.043s)
2022-09-27T16:48:18.4082769Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s)
2022-09-27T16:48:18.4508802Z   test_gather_gpu (__main__.TestDataParallel) ... ok (0.042s)
2022-09-27T16:48:18.4561608Z   test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s)
2022-09-27T16:48:18.4618846Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s)
2022-09-27T16:48:18.4641896Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s)
2022-09-27T16:48:18.4717831Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.007s)
2022-09-27T16:48:18.4761258Z   test_replicate (__main__.TestDataParallel) ... ok (0.004s)
2022-09-27T16:48:18.4795826Z   test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s)
2022-09-27T16:48:18.4829848Z   test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s)
2022-09-27T16:48:18.5012198Z   test_scatter_cpu (__main__.TestDataParallel) ... ok (0.018s)
2022-09-27T16:48:18.5197767Z   test_scatter_gpu (__main__.TestDataParallel) ... ok (0.018s)
2022-09-27T16:48:19.8390872Z   test_strided_grad_layout (__main__.TestDataParallel) ... ok (1.319s)
2022-09-27T16:48:19.8480259Z   test_zero_grad (__main__.TestDataParallel) ... ok (0.009s)
2022-09-27T16:48:19.8527114Z   test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8568505Z   test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8608893Z   test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8654300Z   test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8698719Z   test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8742179Z   test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8787667Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8832540Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8876964Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8922258Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.8966867Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.9011096Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.9056706Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.9101582Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.9147050Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-09-27T16:48:19.9147766Z 
2022-09-27T16:48:19.9148141Z ----------------------------------------------------------------------
2022-09-27T16:48:19.9148497Z Ran 46 tests in 4.908s
2022-09-27T16:48:19.9148669Z 
2022-09-27T16:48:19.9148753Z OK
2022-09-27T16:48:19.9148893Z 
2022-09-27T16:48:19.9149027Z Generating XML reports...
2022-09-27T16:48:19.9218075Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220927164814.xml
2022-09-27T16:48:19.9235543Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220927164814.xml
2022-09-27T16:48:20.9251386Z Running distributed/rpc/test_share_memory ... [2022-09-27 16:48:20.924557]
2022-09-27T16:48:20.9252224Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:20.924667]
2022-09-27T16:48:22.8196850Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:48:22.8197356Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-09-27T16:48:22.8197748Z test_case (__main__.TestRPCPickler)
2022-09-27T16:48:24.4365112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:24.4365610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:24.4367132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:24.4367621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:24.6915675Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-09-27T16:48:24.6930413Z 
2022-09-27T16:48:24.6930920Z Running tests...
2022-09-27T16:48:24.6931729Z ----------------------------------------------------------------------
2022-09-27T16:48:27.9825171Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:27.9825794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:27.9827028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:27.9827499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:29.2593405Z ok (4.566s)
2022-09-27T16:48:29.2595787Z 
2022-09-27T16:48:29.2596535Z ----------------------------------------------------------------------
2022-09-27T16:48:29.2596915Z Ran 1 test in 4.566s
2022-09-27T16:48:29.2597097Z 
2022-09-27T16:48:29.2597204Z OK
2022-09-27T16:48:29.2597358Z 
2022-09-27T16:48:29.2597448Z Generating XML reports...
2022-09-27T16:48:29.2646392Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220927164824.xml
2022-09-27T16:48:30.0596472Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-09-27 16:48:30.059076]
2022-09-27T16:48:30.0597356Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:30.059176]
2022-09-27T16:48:32.0244775Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-09-27T16:48:32.0260806Z 
2022-09-27T16:48:32.0261246Z Running tests...
2022-09-27T16:48:32.0261726Z ----------------------------------------------------------------------
2022-09-27T16:48:33.6553949Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:33.6753732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64789
2022-09-27T16:48:33.6758206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64790
2022-09-27T16:48:33.6765541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64791
2022-09-27T16:48:33.6772708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64792
2022-09-27T16:48:35.3543425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:35.3544899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:35.3545968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:35.3546814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:35.3578366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:35.3579320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:35.3581530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:35.3582415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:35.3601670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:35.3602617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:35.3607317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:35.3608220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:35.3844400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:35.3845669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:35.3847625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:35.3848555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:35.6235432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:35.6272183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-09-27T16:48:35.6304990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:35.6493302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-09-27T16:48:36.1844248Z skip: Need at least 4 CUDA devices (4.158s)
2022-09-27T16:48:36.1844523Z 
2022-09-27T16:48:36.1844938Z ----------------------------------------------------------------------
2022-09-27T16:48:36.1845271Z Ran 1 test in 4.158s
2022-09-27T16:48:36.1845472Z 
2022-09-27T16:48:36.1845587Z OK (skipped=1)
2022-09-27T16:48:36.1845693Z 
2022-09-27T16:48:36.1845801Z Generating XML reports...
2022-09-27T16:48:36.1886273Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220927164832.xml
2022-09-27T16:48:36.6417791Z Running distributed/elastic/utils/util_test ... [2022-09-27 16:48:36.641223]
2022-09-27T16:48:36.6418691Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:36.641316]
2022-09-27T16:48:38.5424062Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-09-27T16:48:38.5440192Z 
2022-09-27T16:48:38.5440428Z Running tests...
2022-09-27T16:48:38.5440883Z ----------------------------------------------------------------------
2022-09-27T16:48:40.2432494Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (1.699s)
2022-09-27T16:48:40.2455175Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-09-27T16:48:40.2485740Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.003s)
2022-09-27T16:48:40.3496766Z   test_get_logger (__main__.UtilTest) ... ok (0.101s)
2022-09-27T16:48:40.3503962Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:48:40.3514017Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:48:40.3528016Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-09-27T16:48:40.3528497Z 
2022-09-27T16:48:40.3529093Z ----------------------------------------------------------------------
2022-09-27T16:48:40.3529386Z Ran 7 tests in 1.809s
2022-09-27T16:48:40.3531236Z 
2022-09-27T16:48:40.3531435Z OK
2022-09-27T16:48:40.3531764Z 
2022-09-27T16:48:40.3532031Z Generating XML reports...
2022-09-27T16:48:40.3569257Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220927164838.xml
2022-09-27T16:48:40.3575630Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220927164838.xml
2022-09-27T16:48:40.7981981Z Running distributed/nn/jit/test_instantiator ... [2022-09-27 16:48:40.797606]
2022-09-27T16:48:40.7982783Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:40.797706]
2022-09-27T16:48:42.7332740Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-09-27T16:48:42.7349519Z 
2022-09-27T16:48:42.7350030Z Running tests...
2022-09-27T16:48:42.7350906Z ----------------------------------------------------------------------
2022-09-27T16:48:44.4517931Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.716s)
2022-09-27T16:48:44.4538760Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-09-27T16:48:44.4633607Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.009s)
2022-09-27T16:48:44.4633974Z 
2022-09-27T16:48:44.4634505Z ----------------------------------------------------------------------
2022-09-27T16:48:44.4634852Z Ran 3 tests in 1.728s
2022-09-27T16:48:44.4635032Z 
2022-09-27T16:48:44.4635071Z OK
2022-09-27T16:48:44.4635192Z 
2022-09-27T16:48:44.4635325Z Generating XML reports...
2022-09-27T16:48:44.4672527Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220927164842.xml
2022-09-27T16:48:44.8965619Z Running distributed/_shard/checkpoint/test_planner ... [2022-09-27 16:48:44.895982]
2022-09-27T16:48:44.8966479Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:44.896081]
2022-09-27T16:48:46.7722475Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_planner
2022-09-27T16:48:46.7738110Z 
2022-09-27T16:48:46.7738389Z Running tests...
2022-09-27T16:48:46.7738837Z ----------------------------------------------------------------------
2022-09-27T16:48:48.4428540Z   test_global_plan (__main__.TestSavePlan) ... ok (1.669s)
2022-09-27T16:48:48.4470447Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2022-09-27T16:48:48.4502424Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2022-09-27T16:48:48.4537541Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s)
2022-09-27T16:48:48.4571614Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2022-09-27T16:48:48.4572313Z 
2022-09-27T16:48:48.4572739Z ----------------------------------------------------------------------
2022-09-27T16:48:48.4573102Z Ran 5 tests in 1.683s
2022-09-27T16:48:48.4573291Z 
2022-09-27T16:48:48.4573377Z OK
2022-09-27T16:48:48.4573508Z 
2022-09-27T16:48:48.4573640Z Generating XML reports...
2022-09-27T16:48:48.4611584Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20220927164846.xml
2022-09-27T16:48:48.8849098Z Running distributed/fsdp/test_utils ... [2022-09-27 16:48:48.884354]
2022-09-27T16:48:48.8850173Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:48.884458]
2022-09-27T16:48:50.7469828Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-09-27T16:48:50.7484085Z 
2022-09-27T16:48:50.7484543Z Running tests...
2022-09-27T16:48:50.7485021Z ----------------------------------------------------------------------
2022-09-27T16:48:52.3888409Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (1.640s)
2022-09-27T16:48:52.3919585Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-09-27T16:48:52.3954015Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s)
2022-09-27T16:48:52.3961081Z   test_packed_sequence (__main__.TestUtils)
2022-09-27T16:48:52.3984498Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-09-27T16:48:52.3995903Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-09-27T16:48:52.3996176Z 
2022-09-27T16:48:52.3996463Z ----------------------------------------------------------------------
2022-09-27T16:48:52.3996828Z Ran 5 tests in 1.651s
2022-09-27T16:48:52.3997003Z 
2022-09-27T16:48:52.3997104Z OK
2022-09-27T16:48:52.3997243Z 
2022-09-27T16:48:52.3997374Z Generating XML reports...
2022-09-27T16:48:52.4036343Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220927164850.xml
2022-09-27T16:48:52.8237748Z Running distributed/elastic/metrics/api_test ... [2022-09-27 16:48:52.823203]
2022-09-27T16:48:52.8238876Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:52.823302]
2022-09-27T16:48:54.7102989Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-09-27T16:48:54.7117522Z 
2022-09-27T16:48:54.7117978Z Running tests...
2022-09-27T16:48:54.7118508Z ----------------------------------------------------------------------
2022-09-27T16:48:56.2859220Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (1.574s)
2022-09-27T16:48:56.2873759Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2022-09-27T16:48:56.3507029Z   test_profile (__main__.MetricsApiTest) ... ok (0.063s)
2022-09-27T16:48:56.3507336Z 
2022-09-27T16:48:56.3507761Z ----------------------------------------------------------------------
2022-09-27T16:48:56.3508098Z Ran 3 tests in 1.639s
2022-09-27T16:48:56.3508274Z 
2022-09-27T16:48:56.3508374Z OK
2022-09-27T16:48:56.3508516Z 
2022-09-27T16:48:56.3508623Z Generating XML reports...
2022-09-27T16:48:56.3546424Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220927164854.xml
2022-09-27T16:48:56.7822546Z Running distributed/_shard/test_replicated_tensor ... [2022-09-27 16:48:56.781694]
2022-09-27T16:48:56.7823365Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:56.781793]
2022-09-27T16:48:58.9300703Z Running distributed/elastic/timer/api_test ... [2022-09-27 16:48:58.929524]
2022-09-27T16:48:58.9301508Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:48:58.929631]
2022-09-27T16:49:00.8100785Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-09-27 16:49:00.809511]
2022-09-27T16:49:00.8101505Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-09-27 16:49:00.809613]
2022-09-27T16:49:03.4017841Z ============================= test session starts ==============================
2022-09-27T16:49:03.4018471Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:03.4147343Z cachedir: .pytest_cache
2022-09-27T16:49:03.4147961Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:03.4148407Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:03.4148742Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:03.4149293Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:03.4630762Z collecting ... 
2022-09-27T16:49:03.4631190Z collected 13 items                                                             
2022-09-27T16:49:03.4636061Z 
2022-09-27T16:49:06.2455827Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED  [  7%]
2022-09-27T16:49:08.2568823Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%]
2022-09-27T16:49:08.3166380Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%]
2022-09-27T16:49:08.3419844Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-09-27T16:49:08.3963809Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%]
2022-09-27T16:49:08.4495682Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%]
2022-09-27T16:49:08.5114032Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%]
2022-09-27T16:49:08.5365794Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-09-27T16:49:08.5883099Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%]
2022-09-27T16:49:08.6392235Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%]
2022-09-27T16:49:08.6906413Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%]
2022-09-27T16:49:08.7055647Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-09-27T16:49:08.7398164Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-09-27T16:49:08.7399379Z 
2022-09-27T16:49:08.7399764Z ======================== 10 passed, 3 skipped in 5.34s =========================
2022-09-27T16:49:09.5804284Z Running distributed/pipeline/sync/skip/test_portal ... [2022-09-27 16:49:09.579843]
2022-09-27T16:49:09.5804997Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-09-27 16:49:09.579947]
2022-09-27T16:49:11.7265115Z ============================= test session starts ==============================
2022-09-27T16:49:11.7265720Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:11.7361542Z cachedir: .pytest_cache
2022-09-27T16:49:11.7362380Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:11.7362854Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:11.7363158Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:11.7363956Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:11.7942834Z collecting ... 
2022-09-27T16:49:11.7943300Z collected 10 items                                                             
2022-09-27T16:49:11.7947901Z 
2022-09-27T16:49:13.1550338Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-09-27T16:49:13.1580199Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-09-27T16:49:13.1598567Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-09-27T16:49:13.1614697Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-09-27T16:49:13.1630750Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-09-27T16:49:13.1646935Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-09-27T16:49:13.1662867Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-09-27T16:49:13.1678993Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-09-27T16:49:13.1695412Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-09-27T16:49:13.1742894Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-09-27T16:49:13.1744895Z 
2022-09-27T16:49:13.1745335Z ============================== 10 passed in 1.45s ==============================
2022-09-27T16:49:13.5839651Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-09-27 16:49:13.583452]
2022-09-27T16:49:13.5840360Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-09-27 16:49:13.583552]
2022-09-27T16:49:15.7429133Z ============================= test session starts ==============================
2022-09-27T16:49:15.7429749Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:15.7525594Z cachedir: .pytest_cache
2022-09-27T16:49:15.7526586Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:15.7527395Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:15.7527701Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:15.7528260Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:15.7743766Z collecting ... 
2022-09-27T16:49:15.7744205Z collected 9 items                                                              
2022-09-27T16:49:15.7748405Z 
2022-09-27T16:49:15.7783523Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-09-27T16:49:15.7801964Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-09-27T16:49:15.7819732Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-09-27T16:49:15.7840573Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-09-27T16:49:15.7859494Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-09-27T16:49:15.7878138Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-09-27T16:49:15.7896079Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-09-27T16:49:15.7917311Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-09-27T16:49:15.7940311Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-09-27T16:49:15.7941773Z 
2022-09-27T16:49:15.7942377Z ============================== 9 passed in 0.05s ===============================
2022-09-27T16:49:16.1054865Z Running distributed/pipeline/sync/test_checkpoint ... [2022-09-27 16:49:16.104929]
2022-09-27T16:49:16.1055537Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-09-27 16:49:16.105077]
2022-09-27T16:49:18.2013183Z ============================= test session starts ==============================
2022-09-27T16:49:18.2013740Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:18.2108916Z cachedir: .pytest_cache
2022-09-27T16:49:18.2109502Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:18.2110186Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:18.2110511Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:18.2111060Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:18.2578420Z collecting ... 
2022-09-27T16:49:18.2578981Z collected 9 items                                                              
2022-09-27T16:49:18.2583615Z 
2022-09-27T16:49:18.2659511Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-09-27T16:49:19.5923888Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-09-27T16:49:19.5944718Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-09-27T16:49:19.5964789Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-09-27T16:49:19.6000331Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-09-27T16:49:19.6034332Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-09-27T16:49:19.6052075Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-09-27T16:49:19.6065999Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-09-27T16:49:19.6088081Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-09-27T16:49:19.6089251Z 
2022-09-27T16:49:19.6089547Z ============================== 9 passed in 1.41s ===============================
2022-09-27T16:49:19.9867409Z Running distributed/pipeline/sync/test_dependency ... [2022-09-27 16:49:19.986196]
2022-09-27T16:49:19.9868066Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-09-27 16:49:19.986301]
2022-09-27T16:49:22.1111449Z ============================= test session starts ==============================
2022-09-27T16:49:22.1112613Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:22.1210370Z cachedir: .pytest_cache
2022-09-27T16:49:22.1212061Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:22.1212976Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:22.1213637Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:22.1214696Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:22.1689399Z collecting ... 
2022-09-27T16:49:22.1690695Z collected 6 items                                                              
2022-09-27T16:49:22.1695842Z 
2022-09-27T16:49:23.5376713Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED      [ 16%]
2022-09-27T16:49:23.5391965Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-09-27T16:49:23.5409082Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-09-27T16:49:23.5427317Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-09-27T16:49:23.5443156Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-09-27T16:49:23.5463144Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-09-27T16:49:23.5464220Z 
2022-09-27T16:49:23.5464832Z ============================== 6 passed in 1.44s ===============================
2022-09-27T16:49:23.9406608Z Running distributed/pipeline/sync/test_phony ... [2022-09-27 16:49:23.940041]
2022-09-27T16:49:23.9407362Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-09-27 16:49:23.940141]
2022-09-27T16:49:26.0844584Z ============================= test session starts ==============================
2022-09-27T16:49:26.0845172Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:26.0941232Z cachedir: .pytest_cache
2022-09-27T16:49:26.0941856Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:26.0942319Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:26.0942617Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:26.0943162Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:26.1121332Z collecting ... 
2022-09-27T16:49:26.1121729Z collected 4 items                                                              
2022-09-27T16:49:26.1126258Z 
2022-09-27T16:49:26.1159892Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-09-27T16:49:26.1177061Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-09-27T16:49:26.1192200Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-09-27T16:49:26.1394025Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-09-27T16:49:26.1395394Z 
2022-09-27T16:49:26.1396033Z ============================== 4 passed in 0.06s ===============================
2022-09-27T16:49:26.4514527Z Running distributed/pipeline/sync/test_stream ... [2022-09-27 16:49:26.450934]
2022-09-27T16:49:26.4515240Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-09-27 16:49:26.451037]
2022-09-27T16:49:28.5879568Z ============================= test session starts ==============================
2022-09-27T16:49:28.5880198Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:49:28.5976377Z cachedir: .pytest_cache
2022-09-27T16:49:28.5977001Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:49:28.5977399Z torch: 1.13.0a0+git52424e2
2022-09-27T16:49:28.5977721Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:49:28.5978268Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:49:28.6534723Z collecting ... 
2022-09-27T16:49:28.6535172Z collected 19 items                                                             
2022-09-27T16:49:28.6539381Z 
2022-09-27T16:49:28.6570212Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-09-27T16:49:30.0198623Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%]
2022-09-27T16:49:30.0211797Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-09-27T16:49:30.0226124Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%]
2022-09-27T16:49:30.0239088Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-09-27T16:49:30.0252745Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%]
2022-09-27T16:49:30.0265807Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-09-27T16:49:30.0279439Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%]
2022-09-27T16:49:30.0292608Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-09-27T16:49:30.0306601Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%]
2022-09-27T16:49:30.0319477Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-09-27T16:49:30.0333441Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%]
2022-09-27T16:49:30.0551918Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-09-27T16:49:30.5046800Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%]
2022-09-27T16:49:30.5064118Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%]
2022-09-27T16:49:30.8530686Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%]
2022-09-27T16:49:30.8545481Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-09-27T16:49:31.1650487Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%]
2022-09-27T16:49:31.1677682Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%]
2022-09-27T16:49:31.1679113Z 
2022-09-27T16:49:31.1679605Z ============================== 19 passed in 2.58s ==============================
2022-09-27T16:49:31.5577866Z Running distributed/rpc/test_faulty_agent ... [2022-09-27 16:49:31.557222]
2022-09-27T16:49:31.5578680Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:49:31.557336]
2022-09-27T16:49:33.4181378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81nyt270
2022-09-27T16:49:33.4182055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81nyt270/_remote_module_non_scriptable.py
2022-09-27T16:49:34.5374640Z 
2022-09-27T16:49:34.5375142Z real	62m51.874s
2022-09-27T16:49:34.5375537Z user	110m48.600s
2022-09-27T16:49:34.5375765Z sys	52m58.993s
2022-09-27T16:49:34.5376022Z + assert_git_not_dirty
2022-09-27T16:49:34.5376537Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-09-27T16:49:34.5377299Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-09-27T16:49:34.5378949Z ++ git status --porcelain
2022-09-27T16:49:36.0428547Z + git_status=
2022-09-27T16:49:36.0429339Z + [[ -n '' ]]
2022-09-27T16:49:36.0429869Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-09-27T16:49:36.0430187Z + [[ 3 == 1 ]]
2022-09-27T16:49:36.0430430Z + [[ 3 == 1 ]]
2022-09-27T16:49:36.0517459Z Prepare all required actions
2022-09-27T16:49:36.0517905Z Getting action download info
2022-09-27T16:49:36.2398313Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-09-27T16:49:36.2398621Z with:
2022-09-27T16:49:36.2399103Z   github-token: ***
2022-09-27T16:49:36.2399329Z env:
2022-09-27T16:49:36.2399583Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:36.2399862Z   GPU_FLAG: --gpus all
2022-09-27T16:49:36.2400247Z ##[endgroup]
2022-09-27T16:49:36.2435755Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-09-27T16:49:36.2436079Z with:
2022-09-27T16:49:36.2436288Z   shell: bash
2022-09-27T16:49:36.2436542Z   timeout_minutes: 10
2022-09-27T16:49:36.2436790Z   max_attempts: 5
2022-09-27T16:49:36.2437025Z   retry_wait_seconds: 30
2022-09-27T16:49:36.2437583Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-09-27T16:49:36.2438096Z   polling_interval_seconds: 1
2022-09-27T16:49:36.2438354Z   warning_on_retry: true
2022-09-27T16:49:36.2438627Z   continue_on_error: false
2022-09-27T16:49:36.2438872Z env:
2022-09-27T16:49:36.2439096Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:36.2439369Z   GPU_FLAG: --gpus all
2022-09-27T16:49:36.2439763Z   GITHUB_TOKEN: ***
2022-09-27T16:49:36.2440018Z ##[endgroup]
2022-09-27T16:49:36.2993378Z 
2022-09-27T16:49:36.3059251Z + python3 -m pip install requests==2.26.0
2022-09-27T16:49:36.6001735Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:49:36.7416498Z Collecting requests==2.26.0
2022-09-27T16:49:36.7632061Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-09-27T16:49:36.8269106Z Collecting idna<4,>=2.5; python_version >= "3"
2022-09-27T16:49:36.8333615Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-09-27T16:49:36.8983838Z Collecting certifi>=2017.4.17
2022-09-27T16:49:36.9031612Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-09-27T16:49:37.0106319Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-09-27T16:49:37.0149200Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-09-27T16:49:37.1314713Z Collecting urllib3<1.27,>=1.21.1
2022-09-27T16:49:37.1365683Z   Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
2022-09-27T16:49:37.2521133Z Installing collected packages: idna, certifi, charset-normalizer, urllib3, requests
2022-09-27T16:49:37.3837487Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:49:37.3838180Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:49:37.5117848Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.12
2022-09-27T16:49:37.5656331Z ++ python3 .github/scripts/get_workflow_job_id.py 3133193930 i-00fe60cd3c4c7e69b
2022-09-27T16:49:41.1743602Z + GHA_WORKFLOW_JOB_ID=8576432567
2022-09-27T16:49:41.1744613Z + echo '::set-output name=job-id::8576432567'
2022-09-27T16:49:41.3087165Z Command completed after 1 attempt(s).
2022-09-27T16:49:41.3087497Z 
2022-09-27T16:49:41.3231641Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:49:41.3231985Z kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:49:41.3245622Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:49:41.3245952Z env:
2022-09-27T16:49:41.3246188Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.3246469Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.3246748Z   MONITOR_SCRIPT_PID: 56687
2022-09-27T16:49:41.3246998Z ##[endgroup]
2022-09-27T16:49:41.3342806Z Prepare all required actions
2022-09-27T16:49:41.3343195Z Getting action download info
2022-09-27T16:49:41.4915363Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-09-27T16:49:41.6372959Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-09-27T16:49:41.6373269Z with:
2022-09-27T16:49:41.6373618Z   file-suffix: test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567
2022-09-27T16:49:41.6373974Z env:
2022-09-27T16:49:41.6374218Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.6374471Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.6374728Z ##[endgroup]
2022-09-27T16:49:41.6407327Z ##[group]Run # Remove any previous test jsons if they exist
2022-09-27T16:49:41.6407871Z # Remove any previous test jsons if they exist
2022-09-27T16:49:41.6408196Z rm -f test-jsons-*.zip
2022-09-27T16:49:41.6408519Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-09-27T16:49:41.6421413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:49:41.6421728Z env:
2022-09-27T16:49:41.6421961Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.6422248Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.6422637Z   FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567
2022-09-27T16:49:41.6422981Z ##[endgroup]
2022-09-27T16:49:41.6578676Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-09-27T16:49:41.6613531Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-09-27T16:49:41.6620844Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-09-27T16:49:41.6621936Z   adding: test/.pytorch-slow-tests.json (deflated 75%)
2022-09-27T16:49:41.6628887Z   adding: test/.pytorch-disabled-tests.json (deflated 85%)
2022-09-27T16:49:41.6654655Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:49:41.6655050Z # Remove any previous test reports if they exist
2022-09-27T16:49:41.6655390Z rm -f test-reports-*.zip
2022-09-27T16:49:41.6655720Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-09-27T16:49:41.6667691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:49:41.6668002Z env:
2022-09-27T16:49:41.6668256Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.6668518Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.6668901Z   FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567
2022-09-27T16:49:41.6669270Z ##[endgroup]
2022-09-27T16:49:41.6782915Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154648.xml (deflated 38%)
2022-09-27T16:49:41.6783623Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154655.xml (deflated 38%)
2022-09-27T16:49:41.6784315Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154703.xml (deflated 38%)
2022-09-27T16:49:41.6784985Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154709.xml (deflated 37%)
2022-09-27T16:49:41.6785653Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154716.xml (deflated 38%)
2022-09-27T16:49:41.6786323Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154724.xml (deflated 38%)
2022-09-27T16:49:41.6786999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154732.xml (deflated 38%)
2022-09-27T16:49:41.6787648Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154738.xml (deflated 38%)
2022-09-27T16:49:41.6788320Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154744.xml (deflated 38%)
2022-09-27T16:49:41.6788986Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154751.xml (deflated 37%)
2022-09-27T16:49:41.6789794Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220927154758.xml (deflated 37%)
2022-09-27T16:49:41.6790498Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154804.xml (deflated 37%)
2022-09-27T16:49:41.6791174Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154810.xml (deflated 37%)
2022-09-27T16:49:41.6791870Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154818.xml (deflated 37%)
2022-09-27T16:49:41.6792564Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154825.xml (deflated 37%)
2022-09-27T16:49:41.6793228Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154833.xml (deflated 38%)
2022-09-27T16:49:41.6794099Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154839.xml (deflated 37%)
2022-09-27T16:49:41.6794854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154847.xml (deflated 38%)
2022-09-27T16:49:41.6795594Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154854.xml (deflated 38%)
2022-09-27T16:49:41.6796334Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154902.xml (deflated 38%)
2022-09-27T16:49:41.6797060Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154908.xml (deflated 38%)
2022-09-27T16:49:41.6797808Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20220927154915.xml (deflated 38%)
2022-09-27T16:49:41.6798623Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154923.xml (deflated 44%)
2022-09-27T16:49:41.6799492Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154931.xml (deflated 44%)
2022-09-27T16:49:41.6800358Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154940.xml (deflated 43%)
2022-09-27T16:49:41.6801218Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154948.xml (deflated 43%)
2022-09-27T16:49:41.6802080Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927154957.xml (deflated 45%)
2022-09-27T16:49:41.6802939Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155006.xml (deflated 45%)
2022-09-27T16:49:41.6803783Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155014.xml (deflated 46%)
2022-09-27T16:49:41.6804658Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155023.xml (deflated 46%)
2022-09-27T16:49:41.6805517Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155031.xml (deflated 44%)
2022-09-27T16:49:41.6806373Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155040.xml (deflated 46%)
2022-09-27T16:49:41.6807210Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155048.xml (deflated 46%)
2022-09-27T16:49:41.6808063Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155057.xml (deflated 44%)
2022-09-27T16:49:41.6808922Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155105.xml (deflated 44%)
2022-09-27T16:49:41.6810277Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155114.xml (deflated 43%)
2022-09-27T16:49:41.6811237Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155120.xml (deflated 44%)
2022-09-27T16:49:41.6812034Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155128.xml (deflated 44%)
2022-09-27T16:49:41.6812820Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155135.xml (deflated 44%)
2022-09-27T16:49:41.6813623Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155141.xml (deflated 45%)
2022-09-27T16:49:41.6814421Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155148.xml (deflated 45%)
2022-09-27T16:49:41.6815291Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155154.xml (deflated 50%)
2022-09-27T16:49:41.6816082Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155202.xml (deflated 42%)
2022-09-27T16:49:41.6816879Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155210.xml (deflated 41%)
2022-09-27T16:49:41.6817669Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155218.xml (deflated 41%)
2022-09-27T16:49:41.6818441Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155226.xml (deflated 41%)
2022-09-27T16:49:41.6819227Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155235.xml (deflated 42%)
2022-09-27T16:49:41.6820023Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155243.xml (deflated 42%)
2022-09-27T16:49:41.6820813Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155250.xml (deflated 42%)
2022-09-27T16:49:41.6821581Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155256.xml (deflated 41%)
2022-09-27T16:49:41.6822363Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155302.xml (deflated 41%)
2022-09-27T16:49:41.6823160Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155309.xml (deflated 44%)
2022-09-27T16:49:41.6823953Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155315.xml (deflated 45%)
2022-09-27T16:49:41.6824719Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155322.xml (deflated 40%)
2022-09-27T16:49:41.6825501Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155328.xml (deflated 41%)
2022-09-27T16:49:41.6826292Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155337.xml (deflated 41%)
2022-09-27T16:49:41.6827113Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155343.xml (deflated 41%)
2022-09-27T16:49:41.6827914Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155349.xml (deflated 41%)
2022-09-27T16:49:41.6828679Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220927155359.xml (deflated 40%)
2022-09-27T16:49:41.6829452Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155408.xml (deflated 39%)
2022-09-27T16:49:41.6830264Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155415.xml (deflated 39%)
2022-09-27T16:49:41.6831033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155423.xml (deflated 39%)
2022-09-27T16:49:41.6831764Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155430.xml (deflated 39%)
2022-09-27T16:49:41.6832523Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155436.xml (deflated 39%)
2022-09-27T16:49:41.6833269Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155443.xml (deflated 39%)
2022-09-27T16:49:41.6834016Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155450.xml (deflated 40%)
2022-09-27T16:49:41.6834805Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155457.xml (deflated 39%)
2022-09-27T16:49:41.6835556Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155507.xml (deflated 40%)
2022-09-27T16:49:41.6836299Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155514.xml (deflated 39%)
2022-09-27T16:49:41.6837042Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155522.xml (deflated 39%)
2022-09-27T16:49:41.6837767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155531.xml (deflated 39%)
2022-09-27T16:49:41.6838514Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155538.xml (deflated 39%)
2022-09-27T16:49:41.6839262Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155544.xml (deflated 39%)
2022-09-27T16:49:41.6840005Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155551.xml (deflated 40%)
2022-09-27T16:49:41.6840721Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155558.xml (deflated 39%)
2022-09-27T16:49:41.6841464Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155604.xml (deflated 39%)
2022-09-27T16:49:41.6842207Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155613.xml (deflated 39%)
2022-09-27T16:49:41.6842947Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155620.xml (deflated 40%)
2022-09-27T16:49:41.6843669Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155627.xml (deflated 39%)
2022-09-27T16:49:41.6844417Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155636.xml (deflated 39%)
2022-09-27T16:49:41.6845159Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155642.xml (deflated 39%)
2022-09-27T16:49:41.6845896Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155649.xml (deflated 39%)
2022-09-27T16:49:41.6846612Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155658.xml (deflated 39%)
2022-09-27T16:49:41.6847352Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155704.xml (deflated 40%)
2022-09-27T16:49:41.6848092Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155711.xml (deflated 40%)
2022-09-27T16:49:41.6848841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155720.xml (deflated 39%)
2022-09-27T16:49:41.6849951Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155727.xml (deflated 39%)
2022-09-27T16:49:41.6850832Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155733.xml (deflated 39%)
2022-09-27T16:49:41.6851570Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155742.xml (deflated 39%)
2022-09-27T16:49:41.6852342Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155748.xml (deflated 39%)
2022-09-27T16:49:41.6853082Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155755.xml (deflated 39%)
2022-09-27T16:49:41.6853884Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155802.xml (deflated 39%)
2022-09-27T16:49:41.6854635Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155812.xml (deflated 39%)
2022-09-27T16:49:41.6855374Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155819.xml (deflated 39%)
2022-09-27T16:49:41.6856113Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155826.xml (deflated 39%)
2022-09-27T16:49:41.6856835Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155835.xml (deflated 39%)
2022-09-27T16:49:41.6857573Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155841.xml (deflated 39%)
2022-09-27T16:49:41.6858315Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155848.xml (deflated 39%)
2022-09-27T16:49:41.6859067Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155858.xml (deflated 39%)
2022-09-27T16:49:41.6859786Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155904.xml (deflated 39%)
2022-09-27T16:49:41.6860525Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155911.xml (deflated 39%)
2022-09-27T16:49:41.6861311Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155918.xml (deflated 39%)
2022-09-27T16:49:41.6862052Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155926.xml (deflated 39%)
2022-09-27T16:49:41.6862768Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155933.xml (deflated 39%)
2022-09-27T16:49:41.6863521Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155940.xml (deflated 40%)
2022-09-27T16:49:41.6864261Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155943.xml (deflated 39%)
2022-09-27T16:49:41.6865003Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155949.xml (deflated 40%)
2022-09-27T16:49:41.6865724Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927155951.xml (deflated 39%)
2022-09-27T16:49:41.6866471Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220927160000.xml (deflated 39%)
2022-09-27T16:49:41.6867181Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160007.xml (deflated 39%)
2022-09-27T16:49:41.6867872Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160009.xml (deflated 39%)
2022-09-27T16:49:41.6868538Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160011.xml (deflated 39%)
2022-09-27T16:49:41.6869273Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160013.xml (deflated 39%)
2022-09-27T16:49:41.6869961Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160016.xml (deflated 38%)
2022-09-27T16:49:41.6870638Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220927160018.xml (deflated 39%)
2022-09-27T16:49:41.6871323Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220927160020.xml (deflated 39%)
2022-09-27T16:49:41.6872025Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220927160024.xml (deflated 41%)
2022-09-27T16:49:41.6872877Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220927160030.xml (deflated 92%)
2022-09-27T16:49:41.6873791Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220927160030.xml (deflated 64%)
2022-09-27T16:49:41.6874550Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160557.xml (deflated 38%)
2022-09-27T16:49:41.6875216Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160606.xml (deflated 38%)
2022-09-27T16:49:41.6875881Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160615.xml (deflated 39%)
2022-09-27T16:49:41.6876543Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160621.xml (deflated 38%)
2022-09-27T16:49:41.6877183Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160630.xml (deflated 38%)
2022-09-27T16:49:41.6877853Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160636.xml (deflated 39%)
2022-09-27T16:49:41.6878513Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160642.xml (deflated 39%)
2022-09-27T16:49:41.6879178Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160649.xml (deflated 38%)
2022-09-27T16:49:41.6879816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160655.xml (deflated 38%)
2022-09-27T16:49:41.6880473Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160703.xml (deflated 37%)
2022-09-27T16:49:41.6881134Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160712.xml (deflated 37%)
2022-09-27T16:49:41.6881792Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160720.xml (deflated 38%)
2022-09-27T16:49:41.6882437Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160726.xml (deflated 38%)
2022-09-27T16:49:41.6883098Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160735.xml (deflated 38%)
2022-09-27T16:49:41.6883757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160743.xml (deflated 39%)
2022-09-27T16:49:41.6884385Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160750.xml (deflated 38%)
2022-09-27T16:49:41.6885040Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160758.xml (deflated 38%)
2022-09-27T16:49:41.6885693Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160806.xml (deflated 37%)
2022-09-27T16:49:41.6886344Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220927160815.xml (deflated 37%)
2022-09-27T16:49:41.6887070Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160823.xml (deflated 37%)
2022-09-27T16:49:41.6887817Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160832.xml (deflated 37%)
2022-09-27T16:49:41.6888491Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160841.xml (deflated 37%)
2022-09-27T16:49:41.6889186Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160850.xml (deflated 38%)
2022-09-27T16:49:41.6890343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160900.xml (deflated 39%)
2022-09-27T16:49:41.6891017Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160909.xml (deflated 37%)
2022-09-27T16:49:41.6891812Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20220927160918.xml (deflated 38%)
2022-09-27T16:49:41.6892572Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160927.xml (deflated 41%)
2022-09-27T16:49:41.6893386Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160937.xml (deflated 41%)
2022-09-27T16:49:41.6894160Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160947.xml (deflated 41%)
2022-09-27T16:49:41.6894962Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927160956.xml (deflated 41%)
2022-09-27T16:49:41.6895755Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161006.xml (deflated 41%)
2022-09-27T16:49:41.6896543Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161015.xml (deflated 41%)
2022-09-27T16:49:41.6897366Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161024.xml (deflated 41%)
2022-09-27T16:49:41.6898144Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161033.xml (deflated 41%)
2022-09-27T16:49:41.6898934Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161042.xml (deflated 41%)
2022-09-27T16:49:41.6899728Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161052.xml (deflated 44%)
2022-09-27T16:49:41.6900516Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161100.xml (deflated 45%)
2022-09-27T16:49:41.6901290Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161109.xml (deflated 43%)
2022-09-27T16:49:41.6902093Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161117.xml (deflated 43%)
2022-09-27T16:49:41.6902882Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161126.xml (deflated 45%)
2022-09-27T16:49:41.6903664Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161135.xml (deflated 45%)
2022-09-27T16:49:41.6904434Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161144.xml (deflated 46%)
2022-09-27T16:49:41.6905216Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161152.xml (deflated 46%)
2022-09-27T16:49:41.6906010Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161201.xml (deflated 44%)
2022-09-27T16:49:41.6906797Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161210.xml (deflated 45%)
2022-09-27T16:49:41.6907630Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161218.xml (deflated 45%)
2022-09-27T16:49:41.6908435Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161227.xml (deflated 44%)
2022-09-27T16:49:41.6909220Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161235.xml (deflated 44%)
2022-09-27T16:49:41.6910010Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161244.xml (deflated 42%)
2022-09-27T16:49:41.6910781Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161253.xml (deflated 41%)
2022-09-27T16:49:41.6911622Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161302.xml (deflated 41%)
2022-09-27T16:49:41.6912408Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161311.xml (deflated 44%)
2022-09-27T16:49:41.6913203Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161321.xml (deflated 44%)
2022-09-27T16:49:41.6913974Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161330.xml (deflated 42%)
2022-09-27T16:49:41.6914760Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161336.xml (deflated 41%)
2022-09-27T16:49:41.6915541Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161346.xml (deflated 41%)
2022-09-27T16:49:41.6916333Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161352.xml (deflated 41%)
2022-09-27T16:49:41.6917103Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161401.xml (deflated 41%)
2022-09-27T16:49:41.6917898Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161411.xml (deflated 42%)
2022-09-27T16:49:41.6918691Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161420.xml (deflated 42%)
2022-09-27T16:49:41.6919475Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161424.xml (deflated 42%)
2022-09-27T16:49:41.6920237Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161428.xml (deflated 42%)
2022-09-27T16:49:41.6921034Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161432.xml (deflated 41%)
2022-09-27T16:49:41.6921816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161436.xml (deflated 41%)
2022-09-27T16:49:41.6922613Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161440.xml (deflated 41%)
2022-09-27T16:49:41.6923398Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161444.xml (deflated 42%)
2022-09-27T16:49:41.6924166Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161454.xml (deflated 41%)
2022-09-27T16:49:41.6924945Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161503.xml (deflated 41%)
2022-09-27T16:49:41.6925737Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161512.xml (deflated 41%)
2022-09-27T16:49:41.6926572Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161522.xml (deflated 41%)
2022-09-27T16:49:41.6927349Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161534.xml (deflated 41%)
2022-09-27T16:49:41.6928134Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161540.xml (deflated 41%)
2022-09-27T16:49:41.6928922Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161546.xml (deflated 42%)
2022-09-27T16:49:41.6930073Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161556.xml (deflated 42%)
2022-09-27T16:49:41.6930976Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161606.xml (deflated 41%)
2022-09-27T16:49:41.6931766Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161615.xml (deflated 42%)
2022-09-27T16:49:41.6932564Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161625.xml (deflated 42%)
2022-09-27T16:49:41.6933349Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161631.xml (deflated 42%)
2022-09-27T16:49:41.6934113Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161638.xml (deflated 41%)
2022-09-27T16:49:41.6934896Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161646.xml (deflated 42%)
2022-09-27T16:49:41.6935685Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161652.xml (deflated 42%)
2022-09-27T16:49:41.6936488Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161702.xml (deflated 42%)
2022-09-27T16:49:41.6937252Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161712.xml (deflated 42%)
2022-09-27T16:49:41.6938033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161736.xml (deflated 44%)
2022-09-27T16:49:41.6938821Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161746.xml (deflated 42%)
2022-09-27T16:49:41.6939608Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161755.xml (deflated 41%)
2022-09-27T16:49:41.6940436Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161802.xml (deflated 41%)
2022-09-27T16:49:41.6941218Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161811.xml (deflated 40%)
2022-09-27T16:49:41.6942021Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161820.xml (deflated 41%)
2022-09-27T16:49:41.6942819Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220927161831.xml (deflated 40%)
2022-09-27T16:49:41.6943598Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161840.xml (deflated 41%)
2022-09-27T16:49:41.6944340Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161847.xml (deflated 41%)
2022-09-27T16:49:41.6945094Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161854.xml (deflated 42%)
2022-09-27T16:49:41.6945852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161856.xml (deflated 41%)
2022-09-27T16:49:41.6946711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161902.xml (deflated 41%)
2022-09-27T16:49:41.6947457Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161909.xml (deflated 41%)
2022-09-27T16:49:41.6948215Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161915.xml (deflated 40%)
2022-09-27T16:49:41.6948966Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161922.xml (deflated 42%)
2022-09-27T16:49:41.6949710Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220927161924.xml (deflated 41%)
2022-09-27T16:49:41.6950548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220927161931.xml (deflated 41%)
2022-09-27T16:49:41.6951321Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161933.xml (deflated 38%)
2022-09-27T16:49:41.6952071Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161941.xml (deflated 38%)
2022-09-27T16:49:41.6952823Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161950.xml (deflated 38%)
2022-09-27T16:49:41.6953571Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927161959.xml (deflated 38%)
2022-09-27T16:49:41.6954297Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162008.xml (deflated 39%)
2022-09-27T16:49:41.6955052Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162018.xml (deflated 39%)
2022-09-27T16:49:41.6955850Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162027.xml (deflated 39%)
2022-09-27T16:49:41.6956601Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162036.xml (deflated 38%)
2022-09-27T16:49:41.6957324Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162044.xml (deflated 38%)
2022-09-27T16:49:41.6958073Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162053.xml (deflated 38%)
2022-09-27T16:49:41.6958815Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162106.xml (deflated 39%)
2022-09-27T16:49:41.6959563Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162115.xml (deflated 39%)
2022-09-27T16:49:41.6960292Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162123.xml (deflated 39%)
2022-09-27T16:49:41.6961033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162132.xml (deflated 39%)
2022-09-27T16:49:41.6961830Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162142.xml (deflated 38%)
2022-09-27T16:49:41.6962571Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162149.xml (deflated 39%)
2022-09-27T16:49:41.6963287Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162159.xml (deflated 39%)
2022-09-27T16:49:41.6964033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220927162212.xml (deflated 39%)
2022-09-27T16:49:41.6964774Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220927162220.xml (deflated 40%)
2022-09-27T16:49:41.6965533Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220927162224.xml (deflated 39%)
2022-09-27T16:49:41.6966298Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220927162231.xml (deflated 92%)
2022-09-27T16:49:41.6967158Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220927162231.xml (deflated 85%)
2022-09-27T16:49:41.6968100Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220927162734.xml (deflated 90%)
2022-09-27T16:49:41.6969099Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220927162734.xml (deflated 73%)
2022-09-27T16:49:41.6970494Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20220927163038.xml (deflated 91%)
2022-09-27T16:49:41.6971322Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220927163243.xml (deflated 84%)
2022-09-27T16:49:41.6972110Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220927163243.xml (deflated 85%)
2022-09-27T16:49:41.6972912Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163531.xml (deflated 40%)
2022-09-27T16:49:41.6973711Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163538.xml (deflated 40%)
2022-09-27T16:49:41.6974526Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163545.xml (deflated 41%)
2022-09-27T16:49:41.6975354Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163551.xml (deflated 40%)
2022-09-27T16:49:41.6976175Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163558.xml (deflated 40%)
2022-09-27T16:49:41.6976977Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163605.xml (deflated 40%)
2022-09-27T16:49:41.6977945Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163612.xml (deflated 41%)
2022-09-27T16:49:41.6978765Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163619.xml (deflated 40%)
2022-09-27T16:49:41.6979574Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220927163626.xml (deflated 40%)
2022-09-27T16:49:41.6980388Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163633.xml (deflated 39%)
2022-09-27T16:49:41.6981180Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163639.xml (deflated 39%)
2022-09-27T16:49:41.6981995Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163648.xml (deflated 39%)
2022-09-27T16:49:41.6982799Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163656.xml (deflated 39%)
2022-09-27T16:49:41.6983602Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220927163705.xml (deflated 39%)
2022-09-27T16:49:41.6984321Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927163717.xml (deflated 39%)
2022-09-27T16:49:41.6985021Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220927163721.xml (deflated 39%)
2022-09-27T16:49:41.6985710Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927163725.xml (deflated 39%)
2022-09-27T16:49:41.6986481Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220927163729.xml (deflated 39%)
2022-09-27T16:49:41.6987198Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927163733.xml (deflated 40%)
2022-09-27T16:49:41.6988160Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220927163737.xml (deflated 40%)
2022-09-27T16:49:41.6988911Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20220927163741.xml (deflated 39%)
2022-09-27T16:49:41.6989635Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927163743.xml (deflated 39%)
2022-09-27T16:49:41.6990435Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220927163747.xml (deflated 39%)
2022-09-27T16:49:41.6991151Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220927163751.xml (deflated 39%)
2022-09-27T16:49:41.6991858Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220927163755.xml (deflated 39%)
2022-09-27T16:49:41.6992559Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927163759.xml (deflated 39%)
2022-09-27T16:49:41.6993276Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220927163803.xml (deflated 39%)
2022-09-27T16:49:41.6993993Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163807.xml (deflated 39%)
2022-09-27T16:49:41.6994709Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163811.xml (deflated 39%)
2022-09-27T16:49:41.6995408Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163815.xml (deflated 39%)
2022-09-27T16:49:41.6996115Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220927163819.xml (deflated 39%)
2022-09-27T16:49:41.6996813Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927163833.xml (deflated 38%)
2022-09-27T16:49:41.6997511Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220927163837.xml (deflated 38%)
2022-09-27T16:49:41.6998179Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163841.xml (deflated 38%)
2022-09-27T16:49:41.6998865Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163845.xml (deflated 38%)
2022-09-27T16:49:41.6999550Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163849.xml (deflated 38%)
2022-09-27T16:49:41.7000237Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163853.xml (deflated 37%)
2022-09-27T16:49:41.7000899Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163857.xml (deflated 37%)
2022-09-27T16:49:41.7001584Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163901.xml (deflated 38%)
2022-09-27T16:49:41.7002265Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163905.xml (deflated 38%)
2022-09-27T16:49:41.7002939Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220927163911.xml (deflated 38%)
2022-09-27T16:49:41.7003818Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220927163915.xml (deflated 79%)
2022-09-27T16:49:41.7004676Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164010.xml (deflated 43%)
2022-09-27T16:49:41.7005703Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164014.xml (deflated 43%)
2022-09-27T16:49:41.7006674Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220927164019.xml (deflated 43%)
2022-09-27T16:49:41.7007858Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164024.xml (deflated 41%)
2022-09-27T16:49:41.7008684Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164034.xml (deflated 42%)
2022-09-27T16:49:41.7009524Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164043.xml (deflated 42%)
2022-09-27T16:49:41.7010850Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164053.xml (deflated 41%)
2022-09-27T16:49:41.7011696Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164103.xml (deflated 41%)
2022-09-27T16:49:41.7012519Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164112.xml (deflated 41%)
2022-09-27T16:49:41.7013404Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164122.xml (deflated 41%)
2022-09-27T16:49:41.7014257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220927164131.xml (deflated 41%)
2022-09-27T16:49:41.7015086Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220927164140.xml (deflated 63%)
2022-09-27T16:49:41.7015978Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220927164140.xml (deflated 84%)
2022-09-27T16:49:41.7016956Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20220927164222.xml (deflated 79%)
2022-09-27T16:49:41.7017750Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220927164301.xml (deflated 37%)
2022-09-27T16:49:41.7018507Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164308.xml (deflated 41%)
2022-09-27T16:49:41.7019319Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164312.xml (deflated 40%)
2022-09-27T16:49:41.7020120Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164316.xml (deflated 40%)
2022-09-27T16:49:41.7020907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220927164320.xml (deflated 41%)
2022-09-27T16:49:41.7021740Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164324.xml (deflated 41%)
2022-09-27T16:49:41.7022590Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164331.xml (deflated 41%)
2022-09-27T16:49:41.7023432Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164341.xml (deflated 41%)
2022-09-27T16:49:41.7024257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220927164347.xml (deflated 41%)
2022-09-27T16:49:41.7025102Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220927164356.xml (deflated 55%)
2022-09-27T16:49:41.7025978Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220927164356.xml (deflated 81%)
2022-09-27T16:49:41.7026923Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220927164356.xml (deflated 80%)
2022-09-27T16:49:41.7027727Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220927164356.xml (deflated 64%)
2022-09-27T16:49:41.7028571Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220927164428.xml (deflated 55%)
2022-09-27T16:49:41.7029442Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20220927164428.xml (deflated 78%)
2022-09-27T16:49:41.7030267Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20220927164456.xml (deflated 75%)
2022-09-27T16:49:41.7031146Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20220927164518.xml (deflated 68%)
2022-09-27T16:49:41.7031952Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220927164538.xml (deflated 75%)
2022-09-27T16:49:41.7032785Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220927164555.xml (deflated 42%)
2022-09-27T16:49:41.7033652Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220927164555.xml (deflated 43%)
2022-09-27T16:49:41.7034575Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20220927164611.xml (deflated 68%)
2022-09-27T16:49:41.7035525Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20220927164611.xml (deflated 42%)
2022-09-27T16:49:41.7036579Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220927164611.xml (deflated 44%)
2022-09-27T16:49:41.7037611Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220927164628.xml (deflated 74%)
2022-09-27T16:49:41.7038486Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220927164642.xml (deflated 54%)
2022-09-27T16:49:41.7039313Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220927164656.xml (deflated 59%)
2022-09-27T16:49:41.7040189Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220927164708.xml (deflated 66%)
2022-09-27T16:49:41.7041028Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220927164708.xml (deflated 78%)
2022-09-27T16:49:41.7041868Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220927164720.xml (deflated 60%)
2022-09-27T16:49:41.7042725Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220927164729.xml (deflated 60%)
2022-09-27T16:49:41.7043532Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220927164738.xml (deflated 59%)
2022-09-27T16:49:41.7044331Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220927164747.xml (deflated 40%)
2022-09-27T16:49:41.7045155Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220927164756.xml (deflated 46%)
2022-09-27T16:49:41.7046001Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20220927164805.xml (deflated 45%)
2022-09-27T16:49:41.7046745Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220927164814.xml (deflated 83%)
2022-09-27T16:49:41.7047543Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220927164814.xml (deflated 90%)
2022-09-27T16:49:41.7048335Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20220927164824.xml (deflated 38%)
2022-09-27T16:49:41.7049195Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220927164832.xml (deflated 43%)
2022-09-27T16:49:41.7050504Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220927164838.xml (deflated 62%)
2022-09-27T16:49:41.7051259Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220927164838.xml (deflated 69%)
2022-09-27T16:49:41.7052008Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220927164842.xml (deflated 63%)
2022-09-27T16:49:41.7052779Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20220927164846.xml (deflated 71%)
2022-09-27T16:49:41.7053482Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220927164850.xml (deflated 68%)
2022-09-27T16:49:41.7054226Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220927164854.xml (deflated 62%)
2022-09-27T16:49:41.7055065Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163401.xml (deflated 41%)
2022-09-27T16:49:41.7055952Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163408.xml (deflated 41%)
2022-09-27T16:49:41.7056812Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163414.xml (deflated 43%)
2022-09-27T16:49:41.7057695Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163417.xml (deflated 44%)
2022-09-27T16:49:41.7058566Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163419.xml (deflated 44%)
2022-09-27T16:49:41.7059437Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163422.xml (deflated 44%)
2022-09-27T16:49:41.7060292Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163446.xml (deflated 41%)
2022-09-27T16:49:41.7061173Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163453.xml (deflated 41%)
2022-09-27T16:49:41.7062094Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163500.xml (deflated 44%)
2022-09-27T16:49:41.7062961Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163502.xml (deflated 44%)
2022-09-27T16:49:41.7063828Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163504.xml (deflated 44%)
2022-09-27T16:49:41.7064679Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163507.xml (deflated 44%)
2022-09-27T16:49:41.7065630Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163401.xml (deflated 43%)
2022-09-27T16:49:41.7091497Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163404.xml (deflated 43%)
2022-09-27T16:49:41.7092359Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163406.xml (deflated 42%)
2022-09-27T16:49:41.7093242Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163416.xml (deflated 42%)
2022-09-27T16:49:41.7094134Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163425.xml (deflated 41%)
2022-09-27T16:49:41.7095189Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163434.xml (deflated 42%)
2022-09-27T16:49:41.7096064Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163446.xml (deflated 43%)
2022-09-27T16:49:41.7096933Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163448.xml (deflated 43%)
2022-09-27T16:49:41.7097807Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163451.xml (deflated 41%)
2022-09-27T16:49:41.7098682Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163500.xml (deflated 42%)
2022-09-27T16:49:41.7099562Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163510.xml (deflated 41%)
2022-09-27T16:49:41.7100428Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20220927163519.xml (deflated 42%)
2022-09-27T16:49:41.7142608Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:49:41.7143015Z # Remove any previous test reports if they exist
2022-09-27T16:49:41.7143340Z rm -f usage-log-*.zip
2022-09-27T16:49:41.7143720Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-09-27T16:49:41.7144096Z # so check to see if the file exists first
2022-09-27T16:49:41.7144405Z if [ -f 'usage_log.txt' ]; then
2022-09-27T16:49:41.7144750Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-09-27T16:49:41.7145020Z fi
2022-09-27T16:49:41.7156922Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:49:41.7157232Z env:
2022-09-27T16:49:41.7157464Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.7157738Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.7158128Z   FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567
2022-09-27T16:49:41.7158509Z ##[endgroup]
2022-09-27T16:49:41.7700439Z   adding: usage_log.txt (deflated 94%)
2022-09-27T16:49:41.7748746Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:49:41.7749060Z with:
2022-09-27T16:49:41.7749358Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:49:41.7749680Z   retention-days: 14
2022-09-27T16:49:41.7749949Z   if-no-files-found: warn
2022-09-27T16:49:41.7750240Z   path: test-jsons-*.zip
2022-09-27T16:49:41.7750512Z   name: artifact
2022-09-27T16:49:41.7750762Z   s3-bucket: gha-artifacts
2022-09-27T16:49:41.7751038Z   region: us-east-1
2022-09-27T16:49:41.7751284Z env:
2022-09-27T16:49:41.7751517Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:41.7751805Z   GPU_FLAG: --gpus all
2022-09-27T16:49:41.7752076Z ##[endgroup]
2022-09-27T16:49:42.1970419Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:49:42.1970895Z With the provided path, there will be 1 file uploaded
2022-09-27T16:49:42.1971545Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:49:42.1981643Z Starting upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:42.3665124Z Finished upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:42.3806037Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:49:42.3806353Z with:
2022-09-27T16:49:42.3806654Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:49:42.3806942Z   retention-days: 14
2022-09-27T16:49:42.3807226Z   if-no-files-found: error
2022-09-27T16:49:42.3807518Z   path: test-reports-*.zip
2022-09-27T16:49:42.3807769Z   name: artifact
2022-09-27T16:49:42.3808038Z   s3-bucket: gha-artifacts
2022-09-27T16:49:42.3808312Z   region: us-east-1
2022-09-27T16:49:42.3808717Z env:
2022-09-27T16:49:42.3808944Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:42.3809224Z   GPU_FLAG: --gpus all
2022-09-27T16:49:42.3809486Z ##[endgroup]
2022-09-27T16:49:42.8006452Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:49:42.8006951Z With the provided path, there will be 1 file uploaded
2022-09-27T16:49:42.8007322Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:49:42.8017563Z Starting upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:43.2267384Z Finished upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:43.2410252Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:49:43.2410574Z with:
2022-09-27T16:49:43.2410873Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:49:43.2411161Z   retention-days: 14
2022-09-27T16:49:43.2411448Z   if-no-files-found: ignore
2022-09-27T16:49:43.2411737Z   path: usage-log-*.zip
2022-09-27T16:49:43.2411993Z   name: artifact
2022-09-27T16:49:43.2412261Z   s3-bucket: gha-artifacts
2022-09-27T16:49:43.2412537Z   region: us-east-1
2022-09-27T16:49:43.2412782Z env:
2022-09-27T16:49:43.2413011Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:43.2413299Z   GPU_FLAG: --gpus all
2022-09-27T16:49:43.2413560Z ##[endgroup]
2022-09-27T16:49:43.6603561Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:49:43.6604042Z With the provided path, there will be 1 file uploaded
2022-09-27T16:49:43.6604411Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:49:43.6614851Z Starting upload of usage-log-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:43.8830929Z Finished upload of usage-log-test-distributed-3-3-linux.8xlarge.nvidia.gpu_8576432567.zip
2022-09-27T16:49:43.8978896Z ##[group]Run set -x
2022-09-27T16:49:43.8979195Z set -x
2022-09-27T16:49:43.8979514Z python3 -m pip install -r requirements.txt
2022-09-27T16:49:43.8979885Z python3 -m pip install boto3==1.19.12
2022-09-27T16:49:43.8980295Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:49:43.8993971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:49:43.8994280Z env:
2022-09-27T16:49:43.8994545Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:49:43.8994799Z   GPU_FLAG: --gpus all
2022-09-27T16:49:43.8995081Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T16:49:43.8995354Z   BRANCH: pull/85462
2022-09-27T16:49:43.8995605Z   TEST_CONFIG: distributed
2022-09-27T16:49:43.8995877Z   SHARD_NUMBER: 3
2022-09-27T16:49:43.8996209Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T16:49:43.8996515Z   PR_NUMBER: 85462
2022-09-27T16:49:43.8996794Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T16:49:43.8997149Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T16:49:43.8997450Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:49:43.8997727Z   TAG: 
2022-09-27T16:49:43.8997975Z   WORKFLOW_ID: 3133193930
2022-09-27T16:49:43.8998423Z   GITHUB_TOKEN: ***
2022-09-27T16:49:43.8998703Z   GHA_WORKFLOW_JOB_ID: 8576432567
2022-09-27T16:49:43.8998977Z ##[endgroup]
2022-09-27T16:49:43.9029786Z + python3 -m pip install -r requirements.txt
2022-09-27T16:49:44.2077149Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:49:44.2922505Z Collecting astunparse
2022-09-27T16:49:44.3098986Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-09-27T16:49:44.3431677Z Collecting expecttest
2022-09-27T16:49:44.3477659Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-09-27T16:49:44.3883144Z Collecting future
2022-09-27T16:49:44.3942515Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-09-27T16:49:46.3413773Z Collecting hypothesis
2022-09-27T16:49:46.3491808Z   Downloading hypothesis-6.54.6-py3-none-any.whl (390 kB)
2022-09-27T16:49:47.1841461Z Collecting numpy
2022-09-27T16:49:47.1912203Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-09-27T16:49:47.5307499Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-09-27T16:49:47.6561593Z Collecting pyyaml
2022-09-27T16:49:47.6617355Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-09-27T16:49:47.6825012Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-09-27T16:49:47.7006317Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-09-27T16:49:47.7634834Z Collecting six
2022-09-27T16:49:47.7688548Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-09-27T16:49:47.8035702Z Collecting types-dataclasses
2022-09-27T16:49:47.8081405Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-09-27T16:49:47.8521662Z Collecting typing_extensions
2022-09-27T16:49:47.8627147Z   Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
2022-09-27T16:49:47.9360800Z Collecting sympy
2022-09-27T16:49:47.9479162Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-09-27T16:49:48.1840635Z Collecting wheel<1.0,>=0.23.0
2022-09-27T16:49:48.1883674Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-09-27T16:49:48.2378274Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-09-27T16:49:48.2420138Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-09-27T16:49:48.2761630Z Collecting exceptiongroup>=1.0.0rc8; python_version < "3.11"
2022-09-27T16:49:48.2915026Z   Downloading exceptiongroup-1.0.0rc9-py3-none-any.whl (12 kB)
2022-09-27T16:49:48.3434157Z Collecting attrs>=19.2.0
2022-09-27T16:49:48.3479529Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-09-27T16:49:48.3959946Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-09-27T16:49:48.3985905Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-09-27T16:49:48.3997379Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-09-27T16:49:48.4230504Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-09-27T16:49:48.4515459Z Collecting mpmath>=0.19
2022-09-27T16:49:48.4594629Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-09-27T16:49:48.4859813Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-09-27T16:49:48.6316783Z Installing collected packages: six, wheel, astunparse, expecttest, future, sortedcontainers, exceptiongroup, attrs, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy
2022-09-27T16:49:48.6734026Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:49:48.6734697Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:49:48.7062481Z     Running setup.py install for future: started
2022-09-27T16:49:49.3789333Z     Running setup.py install for future: finished with status 'done'
2022-09-27T16:49:49.6859428Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:49:49.6860126Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:49:51.6839753Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:49:51.6840760Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:50:00.9344851Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:50:00.9345517Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:50:00.9798186Z Successfully installed astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.0rc9 expecttest-0.1.3 future-0.18.2 hypothesis-6.54.6 mpmath-1.2.1 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.3.0 wheel-0.37.1
2022-09-27T16:50:01.0572208Z + python3 -m pip install boto3==1.19.12
2022-09-27T16:50:01.3489540Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:50:02.2942228Z Collecting boto3==1.19.12
2022-09-27T16:50:02.3136091Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-09-27T16:50:02.3697081Z Collecting jmespath<1.0.0,>=0.7.1
2022-09-27T16:50:02.3754712Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-09-27T16:50:03.5425528Z Collecting botocore<1.23.0,>=1.22.12
2022-09-27T16:50:03.5487134Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-09-27T16:50:03.7750955Z Collecting s3transfer<0.6.0,>=0.5.0
2022-09-27T16:50:03.7791062Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-09-27T16:50:03.7959434Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-09-27T16:50:03.8643905Z Collecting python-dateutil<3.0.0,>=2.1
2022-09-27T16:50:03.8687618Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-09-27T16:50:03.8875913Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-09-27T16:50:04.0704207Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-09-27T16:50:04.9737920Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-09-27T16:50:05.0320302Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:50:16.8744797Z [scribe] Scribe access token not provided, sending report via boto3...
2022-09-27T16:50:16.8745096Z 
2022-09-27T16:50:16.8749910Z ----- Historic stats comparison result ------
2022-09-27T16:50:16.8750356Z 
2022-09-27T16:50:16.8750922Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T16:50:16.8751558Z     commit: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:50:16.8751768Z 
2022-09-27T16:50:16.8752034Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-09-27T16:50:16.8753900Z 
2022-09-27T16:50:16.8754390Z     : (master)
2022-09-27T16:50:16.8754674Z     |
2022-09-27T16:50:16.8754966Z     | * 52424e2bf3 (HEAD)            total time  2826.50s
2022-09-27T16:50:16.8755272Z     | |
2022-09-27T16:50:16.8755482Z     | : (4 commits)
2022-09-27T16:50:16.8755717Z     |/
2022-09-27T16:50:16.8756383Z     * c7c2578f93 (base)   9 reports, total time  3338.43s ± 1882.56s
2022-09-27T16:50:16.8756832Z     * 99ad8a3048          9 reports, total time  3402.28s ± 1860.32s
2022-09-27T16:50:16.8757272Z     * 34296e2f4c          9 reports, total time  3340.35s ± 1886.81s
2022-09-27T16:50:16.8757709Z     * 4523ac7aa1          9 reports, total time  3366.30s ± 1853.68s
2022-09-27T16:50:16.8758154Z     * f21e77d9a6          9 reports, total time  3413.29s ± 1861.08s
2022-09-27T16:50:16.8758562Z     * 26a861cb27          9 reports, total time  3325.52s ± 1844.86s
2022-09-27T16:50:16.8758994Z     * 56a41b5998          9 reports, total time  3452.70s ± 1976.19s
2022-09-27T16:50:16.8759474Z     * 1910c5847e          9 reports, total time  3444.18s ± 2043.46s
2022-09-27T16:50:16.8759903Z     * caa0ab557d          9 reports, total time  3312.06s ± 1824.07s
2022-09-27T16:50:16.8760537Z     * 0336308be5          0 reports
2022-09-27T16:50:16.8760795Z     |
2022-09-27T16:50:16.8760994Z     :
2022-09-27T16:50:16.8761151Z 
2022-09-27T16:50:16.8761325Z Removed  (across  894 suites)     0 tests, totaling     0.00s
2022-09-27T16:50:16.8761690Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-09-27T16:50:16.8762053Z Added    (across   75 suites)   705 tests, totaling +2826.50s
2022-09-27T16:50:16.9364545Z Prepare all required actions
2022-09-27T16:50:16.9392027Z ##[group]Run ./.github/actions/teardown-linux
2022-09-27T16:50:16.9392321Z with:
2022-09-27T16:50:16.9392522Z env:
2022-09-27T16:50:16.9392773Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:50:16.9393053Z   GPU_FLAG: --gpus all
2022-09-27T16:50:16.9393291Z ##[endgroup]
2022-09-27T16:50:16.9412999Z ##[group]Run set -eou pipefail
2022-09-27T16:50:16.9413320Z set -eou pipefail
2022-09-27T16:50:16.9413586Z 
2022-09-27T16:50:16.9413900Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-09-27T16:50:16.9414270Z for _ in $(seq 1440); do
2022-09-27T16:50:16.9414581Z     # Break if no ssh session exists anymore
2022-09-27T16:50:16.9414869Z     if [ "$(who)" = "" ]; then
2022-09-27T16:50:16.9415137Z       break
2022-09-27T16:50:16.9415381Z     fi
2022-09-27T16:50:16.9415627Z     echo "."
2022-09-27T16:50:16.9415902Z     sleep 5
2022-09-27T16:50:16.9416144Z done
2022-09-27T16:50:16.9429512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:50:16.9429824Z env:
2022-09-27T16:50:16.9430077Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:50:16.9430335Z   GPU_FLAG: --gpus all
2022-09-27T16:50:16.9430598Z ##[endgroup]
2022-09-27T16:50:16.9460761Z Holding runner for 2 hours until all ssh sessions have logged out
2022-09-27T16:50:16.9562968Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:50:16.9563417Z # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:50:16.9563767Z # shellcheck disable=SC2046
2022-09-27T16:50:16.9564120Z docker stop $(docker ps -q) || true
2022-09-27T16:50:16.9564442Z # Prune all of the docker images
2022-09-27T16:50:16.9564748Z docker system prune -af
2022-09-27T16:50:16.9577463Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:50:16.9577796Z env:
2022-09-27T16:50:16.9578056Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:50:16.9578317Z   GPU_FLAG: --gpus all
2022-09-27T16:50:16.9578583Z ##[endgroup]
2022-09-27T16:50:17.9656024Z 4d347b8a91a0
2022-09-27T16:50:18.7217511Z Deleted Containers:
2022-09-27T16:50:18.7217960Z 4d347b8a91a0d81ea51e22a5c6247010f4ea86e52f35fb357941dd928b7dc4b6
2022-09-27T16:50:18.7218223Z 
2022-09-27T16:50:23.9584769Z Deleted Images:
2022-09-27T16:50:23.9586417Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T16:50:23.9587454Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:9bb261bc4d8aeb82a71b1f0709da9c979e85a12a79c4a85c3fe3adddddcb2663
2022-09-27T16:50:23.9588139Z deleted: sha256:1565775a6d0c052a41180f67487ffe62db8903a6db8b459487e830a767b885e7
2022-09-27T16:50:23.9588558Z deleted: sha256:3e4cb2c2b5f9e2c80e23a8d896fa974adc4b0e3a54cc06c0a8afb922880fdac8
2022-09-27T16:50:23.9589020Z deleted: sha256:d0ad3421a88e79ceec8792dd7f305a7f9da57daa0119b35d0a37114fd2a8dcf9
2022-09-27T16:50:23.9589519Z deleted: sha256:701e765e83a6966eae0a6dd8fc686dd1787c602cf538ef7dea4100368068fcc9
2022-09-27T16:50:23.9589942Z deleted: sha256:8e85fd60215cbc7be6e9f6423e1f87f0e07c0c672606d79f499dbc625e3eda75
2022-09-27T16:50:23.9590394Z deleted: sha256:e5bde08e611f82f97b690c9fa678975e18d51c1d1bc1c8ddef0588f1a022d639
2022-09-27T16:50:23.9590847Z deleted: sha256:68800e2bce8407a9b0b64d467c217a2bb27e85fd2f3ab0c5793759d5443bf962
2022-09-27T16:50:23.9591780Z deleted: sha256:2457e2e8215ddc219f679a7f957fdc3c639147fc569a0cc1bc9f3a3a97ecd0b2
2022-09-27T16:50:23.9592214Z deleted: sha256:b9c61d9be1eb46057ce27c51ac051ba0ba53d440bda828612cfc06ae78352d7a
2022-09-27T16:50:23.9592762Z deleted: sha256:73dd107fbbff1e853191e9dcb9b75847329af3c80a5883c3ae039e14fa4caa0a
2022-09-27T16:50:23.9593336Z deleted: sha256:fe97c2c62ffb15e65ed2751c4c9069d62e91f204ff1c30980d18680617c5ae40
2022-09-27T16:50:23.9593877Z deleted: sha256:3393d072706e15983b6dba8491dcfaf03de10df9be7d9df9a80ef0dedf384b3e
2022-09-27T16:50:23.9594331Z deleted: sha256:328eaea416a7110b96dd1c4421ae76dac29d151937f13f647291840207613cfd
2022-09-27T16:50:23.9594777Z deleted: sha256:1f62c29a7ac809a8add06b0bc14387e38ec12e88034824daccfdd78a0ee24d00
2022-09-27T16:50:23.9595233Z deleted: sha256:a364f929327c0e6baf5da371c2e514c56a7dc2fcbd3f7f2229d49bf2fb3a2f27
2022-09-27T16:50:23.9595664Z deleted: sha256:14a20e2c555e8aea13a33ca9c7d20333ce850c23d3a6726d7b102a426c2bb100
2022-09-27T16:50:23.9596157Z deleted: sha256:865844d2dbf2eb79ca92ace32b76fd966b142ecccd91a6190a256f4d278fed74
2022-09-27T16:50:23.9596608Z deleted: sha256:6a56212e72bab94d436ef418303481e278e79a012bf88cc727231093221416e7
2022-09-27T16:50:23.9597045Z deleted: sha256:8ff11e3f91c81bfe67a7da7b03306cdc936f730f44e7dbc401299c81679f3a20
2022-09-27T16:50:23.9597465Z deleted: sha256:e3ebc0052e1300157e36aa327172ad1098717918585d1494c4efc4afdbb8ecbe
2022-09-27T16:50:23.9597913Z deleted: sha256:4b7db58c33bbdbe2e491f721c5a9724ef07d9a720d77bd517c1f5c8fad212ba2
2022-09-27T16:50:23.9598374Z deleted: sha256:1e68b2d3e3c7bf741f5f060b3a88095db0d5eefd841ce4e86b3c466f882997e6
2022-09-27T16:50:23.9598804Z deleted: sha256:4da2a66a2b27b034b75aa5a024cf7b8cfe1f8762b0725921f8b00cb3a0505759
2022-09-27T16:50:23.9599264Z deleted: sha256:37c6c713aea14d8a4a5f0dfd63f80f04633b0d6f8e7baed0a1feba47d709cbeb
2022-09-27T16:50:23.9599717Z deleted: sha256:c636e275b4e8c2d9022e72ee8d8528006ef92ad2fab903ba909244c2b9aa4bb0
2022-09-27T16:50:23.9600169Z deleted: sha256:2b912a242a1b69ebde3bf40dfab4b693cfa93f9bcb0d8be6c698b44b4284a70d
2022-09-27T16:50:23.9600613Z deleted: sha256:e4ff425d4caf55cc2f5939271ab53c4382b45c5454ccb0e4fa62cbb730aa8ba7
2022-09-27T16:50:23.9601069Z deleted: sha256:f80b712f0269de04dedd4bb68f0a2103eb0ac5bb70e5da74a7fe0544c8a678af
2022-09-27T16:50:23.9601517Z deleted: sha256:c048927fdfe44aec1063f37c0137ad63b7537c23dd3051ad691232ce363048fa
2022-09-27T16:50:23.9601935Z deleted: sha256:85d0da5e4c90646a2049cf17e470deedd05b19ab1535e4932940055fc36e1c91
2022-09-27T16:50:23.9602381Z deleted: sha256:a94964edf2ce4a0b440f5d000d2ea97c6145adcb1ad201ade6214f02cd47626c
2022-09-27T16:50:23.9602826Z deleted: sha256:fc49e4f76476298cd45a4ef31767534bd2b6663c90683ccbb4d911ea77a93d56
2022-09-27T16:50:23.9603270Z deleted: sha256:25a90c74f1ddefff551a5575bbaeaf1fd45c71d629ef061f41a227f2813ebf62
2022-09-27T16:50:23.9603690Z deleted: sha256:9c67a539cd718b76768a481e6313606489c7afbdd2ed5f0d35f94a0c8161ff59
2022-09-27T16:50:23.9604126Z deleted: sha256:78cfd7e90437d18d294dd32f35632e1e0f4e93f98d2f74562b7a2a483d89e847
2022-09-27T16:50:23.9604591Z deleted: sha256:feccb37688a9f8b307ab298febd901ccc8920a2ec8e1660a0eb8de5b6e41a3b1
2022-09-27T16:50:23.9605023Z deleted: sha256:eeded198b25e05f7ca35680574d2863e48240eb544a4945d562943ed7c519eaa
2022-09-27T16:50:23.9605460Z deleted: sha256:eba0a8e367727026cab4d61ce7412eac92df541fa66418907501f777a611d01a
2022-09-27T16:50:23.9605897Z deleted: sha256:ab3963854dae2a8321e764d73ecfb89e77618f391cbe2698ab09c8129a35eb29
2022-09-27T16:50:23.9606323Z deleted: sha256:db198971615e367404b8c928ff87c4bc7e81b060fa78a8431f2124d37e546a69
2022-09-27T16:50:23.9606727Z deleted: sha256:116af78f4b87e494b37b6a65d9e3abdd828764632b0adf869c40db8a8afb7745
2022-09-27T16:50:23.9607161Z deleted: sha256:21479387e93277b4c3c8df1cbd39208f3c6aa4128b6f0eebaafa7e3b82d8a23c
2022-09-27T16:50:23.9607602Z deleted: sha256:5731c54a72f20a93dd164dc484ecd3fd0bf0f0cb98eec92e2382f99c0c43e1a6
2022-09-27T16:50:23.9608027Z deleted: sha256:8b6f381bcc3be3c07a12a8e528c8926c369875dccb5d18323007a3beb34cdf52
2022-09-27T16:50:23.9608550Z deleted: sha256:07dcc550108cf9d433a453ce3e351ac30e67c1ccbc5f7bc71b408e2fb4bcc6a9
2022-09-27T16:50:23.9608992Z deleted: sha256:129bdb873e79117f4e90135f0c6a58f775fcf596f4eb514b803771cef2da8278
2022-09-27T16:50:23.9609452Z deleted: sha256:2d49e3a81bd436bfd20fb4a849cdc98da82cb74afef3de38dda7a946d3fc4153
2022-09-27T16:50:23.9610164Z deleted: sha256:0ba4e259108e5311ddf6b79ae3a35f8f16a4004ef8817e50427baa3cc90ac081
2022-09-27T16:50:23.9610709Z deleted: sha256:c164403226561914f16becdeca65c54d20dba8dad414b062efc34c05c47bf725
2022-09-27T16:50:23.9611160Z deleted: sha256:cbe4006b2e6286d50c1b292fb71b69d5299d65f055285519eafc41eac3ef8a3c
2022-09-27T16:50:23.9611584Z deleted: sha256:edcec18dceb25f1a03ec20de4676464613e69072875a83f5c45e45a31aafc5b9
2022-09-27T16:50:23.9612018Z deleted: sha256:13c4f317ac4bb48997302756b8d5f8b602e835607c9806a1a5b200e9a0657d8a
2022-09-27T16:50:23.9612439Z deleted: sha256:57f043e380f4586c76968d6e062b50bac55254a5be7e80bea3c027a5bb316469
2022-09-27T16:50:23.9612860Z deleted: sha256:3e549931e0240b9aac25dc79ed6a6259863879a5c9bd20755f77cac27c1ab8c8
2022-09-27T16:50:23.9613088Z 
2022-09-27T16:50:23.9710411Z Total reclaimed space: 19.33GB
2022-09-27T16:50:23.9776972Z Post job cleanup.
2022-09-27T16:50:23.9816330Z Post job cleanup.
2022-09-27T16:50:24.1191272Z [command]/usr/bin/git version
2022-09-27T16:50:24.1242094Z git version 2.37.1
2022-09-27T16:50:24.1307555Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3bc4810c-e571-4763-9ceb-b23723e4f5f7' before making global git config changes
2022-09-27T16:50:24.1308151Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T16:50:24.1316952Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T16:50:24.1367016Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T16:50:24.1407115Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T16:50:24.1747750Z Entering 'android/libs/fbjni'
2022-09-27T16:50:24.1791149Z Entering 'third_party/FP16'
2022-09-27T16:50:24.1836659Z Entering 'third_party/FXdiv'
2022-09-27T16:50:24.1878220Z Entering 'third_party/NNPACK'
2022-09-27T16:50:24.1920986Z Entering 'third_party/QNNPACK'
2022-09-27T16:50:24.1965890Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:50:24.2010920Z Entering 'third_party/XNNPACK'
2022-09-27T16:50:24.2067447Z Entering 'third_party/benchmark'
2022-09-27T16:50:24.2111784Z Entering 'third_party/cpuinfo'
2022-09-27T16:50:24.2153924Z Entering 'third_party/cub'
2022-09-27T16:50:24.2197820Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:50:24.2247324Z Entering 'third_party/cutlass'
2022-09-27T16:50:24.2297467Z Entering 'third_party/eigen'
2022-09-27T16:50:24.2342773Z Entering 'third_party/fbgemm'
2022-09-27T16:50:24.2385179Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:50:24.2428946Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:50:24.2472667Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:50:24.2515849Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:50:24.2558968Z Entering 'third_party/flatbuffers'
2022-09-27T16:50:24.2604982Z Entering 'third_party/fmt'
2022-09-27T16:50:24.2649296Z Entering 'third_party/foxi'
2022-09-27T16:50:24.2693545Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:50:24.2737498Z Entering 'third_party/gloo'
2022-09-27T16:50:24.2781451Z Entering 'third_party/googletest'
2022-09-27T16:50:24.2824686Z Entering 'third_party/ideep'
2022-09-27T16:50:24.2866881Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:50:24.2911485Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:50:24.2961697Z Entering 'third_party/ios-cmake'
2022-09-27T16:50:24.3004478Z Entering 'third_party/ittapi'
2022-09-27T16:50:24.3048812Z Entering 'third_party/kineto'
2022-09-27T16:50:24.3090476Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:50:24.3133081Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:50:24.3177181Z Entering 'third_party/nccl/nccl'
2022-09-27T16:50:24.3220046Z Entering 'third_party/neon2sse'
2022-09-27T16:50:24.3262262Z Entering 'third_party/nlohmann'
2022-09-27T16:50:24.3306033Z Entering 'third_party/onnx'
2022-09-27T16:50:24.3362096Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:50:24.3404730Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:50:24.3449209Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:50:24.3490878Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:50:24.3540049Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:50:24.3583808Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:50:24.3626429Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:50:24.3675742Z Entering 'third_party/pocketfft'
2022-09-27T16:50:24.3720068Z Entering 'third_party/protobuf'
2022-09-27T16:50:24.3767640Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:50:24.3811890Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:50:24.3859016Z Entering 'third_party/psimd'
2022-09-27T16:50:24.3900931Z Entering 'third_party/pthreadpool'
2022-09-27T16:50:24.3943127Z Entering 'third_party/pybind11'
2022-09-27T16:50:24.3986377Z Entering 'third_party/python-enum'
2022-09-27T16:50:24.4028856Z Entering 'third_party/python-peachpy'
2022-09-27T16:50:24.4072092Z Entering 'third_party/python-six'
2022-09-27T16:50:24.4114306Z Entering 'third_party/sleef'
2022-09-27T16:50:24.4157075Z Entering 'third_party/tbb'
2022-09-27T16:50:24.4201510Z Entering 'third_party/tensorpipe'
2022-09-27T16:50:24.4244804Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:50:24.4287177Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:50:24.4329967Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:50:24.4373474Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:50:24.4415912Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:50:24.4460670Z Entering 'third_party/zstd'
2022-09-27T16:50:24.4527191Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T16:50:24.4557530Z http.https://github.com/.extraheader
2022-09-27T16:50:24.4568600Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-09-27T16:50:24.4607975Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T16:50:24.4934336Z Entering 'android/libs/fbjni'
2022-09-27T16:50:24.4959407Z http.https://github.com/.extraheader
2022-09-27T16:50:24.4993295Z Entering 'third_party/FP16'
2022-09-27T16:50:24.5018938Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5051970Z Entering 'third_party/FXdiv'
2022-09-27T16:50:24.5076900Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5110151Z Entering 'third_party/NNPACK'
2022-09-27T16:50:24.5135632Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5168388Z Entering 'third_party/QNNPACK'
2022-09-27T16:50:24.5193129Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5226021Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:50:24.5251359Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5283742Z Entering 'third_party/XNNPACK'
2022-09-27T16:50:24.5309337Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5354751Z Entering 'third_party/benchmark'
2022-09-27T16:50:24.5379870Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5413257Z Entering 'third_party/cpuinfo'
2022-09-27T16:50:24.5437904Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5471213Z Entering 'third_party/cub'
2022-09-27T16:50:24.5496765Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5530449Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:50:24.5555192Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5594954Z Entering 'third_party/cutlass'
2022-09-27T16:50:24.5619815Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5659727Z Entering 'third_party/eigen'
2022-09-27T16:50:24.5684804Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5720386Z Entering 'third_party/fbgemm'
2022-09-27T16:50:24.5747062Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5780116Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:50:24.5805520Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5838828Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:50:24.5864891Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5898046Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:50:24.5922682Z http.https://github.com/.extraheader
2022-09-27T16:50:24.5955806Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:50:24.5980791Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6015162Z Entering 'third_party/flatbuffers'
2022-09-27T16:50:24.6040294Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6075249Z Entering 'third_party/fmt'
2022-09-27T16:50:24.6101677Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6134304Z Entering 'third_party/foxi'
2022-09-27T16:50:24.6159165Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6192908Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:50:24.6218171Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6250452Z Entering 'third_party/gloo'
2022-09-27T16:50:24.6275345Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6308494Z Entering 'third_party/googletest'
2022-09-27T16:50:24.6335112Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6367513Z Entering 'third_party/ideep'
2022-09-27T16:50:24.6393021Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6424813Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:50:24.6449752Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6486164Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:50:24.6512556Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6554736Z Entering 'third_party/ios-cmake'
2022-09-27T16:50:24.6580957Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6613539Z Entering 'third_party/ittapi'
2022-09-27T16:50:24.6638261Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6671653Z Entering 'third_party/kineto'
2022-09-27T16:50:24.6697624Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6731690Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:50:24.6756033Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6789182Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:50:24.6814036Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6848147Z Entering 'third_party/nccl/nccl'
2022-09-27T16:50:24.6873716Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6907548Z Entering 'third_party/neon2sse'
2022-09-27T16:50:24.6932818Z http.https://github.com/.extraheader
2022-09-27T16:50:24.6964457Z Entering 'third_party/nlohmann'
2022-09-27T16:50:24.6990614Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7025115Z Entering 'third_party/onnx'
2022-09-27T16:50:24.7050066Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7096556Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:50:24.7121721Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7155208Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:50:24.7180183Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7214977Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:50:24.7239724Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7272126Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:50:24.7298615Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7337169Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:50:24.7361792Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7395207Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:50:24.7420456Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7453385Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:50:24.7478017Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7516127Z Entering 'third_party/pocketfft'
2022-09-27T16:50:24.7541968Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7574505Z Entering 'third_party/protobuf'
2022-09-27T16:50:24.7599329Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7636104Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:50:24.7662868Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7695904Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:50:24.7721414Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7757954Z Entering 'third_party/psimd'
2022-09-27T16:50:24.7782990Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7816089Z Entering 'third_party/pthreadpool'
2022-09-27T16:50:24.7841504Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7874503Z Entering 'third_party/pybind11'
2022-09-27T16:50:24.7900512Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7933904Z Entering 'third_party/python-enum'
2022-09-27T16:50:24.7958690Z http.https://github.com/.extraheader
2022-09-27T16:50:24.7991280Z Entering 'third_party/python-peachpy'
2022-09-27T16:50:24.8016811Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8048877Z Entering 'third_party/python-six'
2022-09-27T16:50:24.8074767Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8107964Z Entering 'third_party/sleef'
2022-09-27T16:50:24.8133006Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8165388Z Entering 'third_party/tbb'
2022-09-27T16:50:24.8191381Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8226597Z Entering 'third_party/tensorpipe'
2022-09-27T16:50:24.8251856Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8284395Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:50:24.8309662Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8342907Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:50:24.8367123Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8399670Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:50:24.8424757Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8458287Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:50:24.8482262Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8514258Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:50:24.8539046Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8575394Z Entering 'third_party/zstd'
2022-09-27T16:50:24.8599863Z http.https://github.com/.extraheader
2022-09-27T16:50:24.8950317Z Cleaning up orphan processes